Cybergis and teragrid science gateways update
1 / 25

CyberGIS and TeraGrid Science Gateways update - PowerPoint PPT Presentation

  • Uploaded on

CyberGIS and TeraGrid Science Gateways update. Nancy Wilkins-Diehr TeraGrid Area Director for Science Gateways [email protected] CyberGIS !. SGW funds a Cyberinfrastructure in GIS workshop in conjunction with the UCGIS meeting in DC in February, 2010

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'CyberGIS and TeraGrid Science Gateways update' - brandi

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Cybergis and teragrid science gateways update

CyberGIS andTeraGrid Science Gateways update

Nancy Wilkins-Diehr

TeraGrid Area Director for Science Gateways

[email protected]

TeraGrid Quarterly, December 7, 2010


  • SGW funds a Cyberinfrastructure in GIS workshop in conjunction with the UCGIS meeting in DC in February, 2010

    • Co-led by Shaowen Wang at NCSA and Nancy

    • Approved by UCGIS board after a lengthy voting process

    • Expected outcome of the workshop – all are happening!

      • Increased visibility

      • New partnerships for TeraGrid, UCGIS, and other pertinent organizations

      • Workshop report

      • Interesting collaborative proposal ideas

      • Potential future publications

    • 50 attendees from throughout the US, Sweden and Australia

    • Workshop attendees also attended UCGIS meeting, including briefing from high-ranking administration officials

TeraGrid Quarterly, December 7, 2010

Shaowen and nancy represent teragrid congressional reading room library of congress
Shaowen and Nancy represent TeraGridCongressional Reading Room, Library of Congress

TeraGrid Quarterly, December 7, 2010

A dministrative officials in attendance
Administrative officials in attendance

  • Karen Siderelis

    • GIO Department of Interior, Acting Chair Federal Geographic Data Committee

  • Raphael Bostic

    • Assistant Secretary for Policy Development and Research, HUD

  • Derek Douglas

    • Special Assistant to the President for Urban Affairs

  • Jerry Johnston

    • CIO, EPA

  • Stephen Lowe

    • GIO, Dept of Agriculture

TeraGrid Quarterly, December 7, 2010

Workshop brings results
Workshop brings results

  • Scientific Software Integration (SSI) award for CyberGIS

    • Shaowen Wang, PI

    • Largest SSI awarded in FY10

      • $4.4M, 10/1/10-9/30/15

    • Academia, government, industry, international partnerships

      • Arizona State University

      • The Computer Network Information Center of the Chinese Academy of Sciences

      • Environmental Systems Research Institute (ESRI)

      • Georgia Institute of Technology

      • Oak Ridge National Laboratory

      • University College London Centre for Advanced Spatial Analysis (England)

      • University Consortium for Geographic Information Science

      • University of California-San Diego

      • University of California-Santa Barbara

      • University of Illinois at Urbana-Champaign

      • University of Washington

      • U.S. Geological Survey

      • Victorian Partnership for Advanced Computing (Australia)

TeraGrid Quarterly, December 7, 2010

Goals and objectives
Goals and objectives

  • Establish CyberGIS as a fundamentally new software framework

    • Integration of CI, GIS, and spatial analysis/modelingcapabilities

    • Widespread scientific breakthroughs, broad societal impacts

  • Participatory evolution of CyberGIS community requirements

  • CyberGIS software integration roadmap

  • High performance and scalable CyberGIS

  • Online CyberGIS gateway

  • CyberGIS testing and integration with national and international CI

  • Community-based and application-driven evaluation of CyberGIS

TeraGrid Quarterly, December 7, 2010

Cybergis vision
CyberGIS vision

Wang, S. 2010. "A CyberGIS Framework for the Synthesis of Cyberinfrastructure, GIS, and Spatial Analysis.“

Annals of the Association of American Geographers, 100(3): 535-557.

TeraGrid Quarterly, December 7, 2010

Initial software components
Initial software components

  • GISolve

    • Large-scale spatial analysis and modeling (SAM), web interface

  • GeoDa/PySAL

    • Spatial data analysis: exploratory data analysis (EDA), exploratory spatial data analysis (ESDA), maximum likelihood (ML) spatial regression

    • Open source library for spatial analysis: weights, computational geometry, ESDA, spatial econometrics, clustering and spatial dynamics

  • Open-Topography

    • Community access to high-resolution, Earth science-oriented, topography data, and related tools and resources


    • On-line tools for expanding public participation in transportation improvement programming

  • pd-GRASS

    • GRASS (Geographic Resources Analysis Support System)

      • Geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization

    • pd-GRASS

      • Shell scripts that distribute display functions of GRASS GIS through the network among several physical monitors in a synchronized manner

TeraGrid Quarterly, December 7, 2010

Early application targets
Early application targets

  • Emergency management

    • Fire, flood, disease, earthquake

    • Managing during the incident and post incident, tracking victims, etc.

    • Real-time integration of data from multiple sources

    • Combining distinct analysis tools via workflows, incorporating HPC when warranted (“computational intensity maps”)

  • Distributed analysis support, collaboration and participation tools through an online service

  • Proof of concept research

TeraGrid Quarterly, December 7, 2010

What have the gateways been up to
What have the gateways been up to?

  • Record high 896 users of community accounts on TG

    • 31% of all users charging jobs

    • 645 users of CIPRES gateway

      • Cited in at least 35 publications including Nature, PNAS, and Cell

      • 77% of all jobs have been submitted from the US, including top-tier institutions such as Harvard, Yale, and Stanford

      • Jobs received regularly from 17 EPSCOR states

      • Job submissions from 34 countries on 5 continents

      • At least 5 undergraduate classes known to use the portal routinely. This is likely an underestimate (based on Web log patterns).

  • International representation

    • Matthew Woitaszek gives keynote address at the International Workshop on Science Gateways (IWSG) in Catania, Italy

    • Rion Dooleyalso in attendance

  • GRAM5

    • This deployment is the final piece needed to support attribute-based authentication

    • Much testing by gateway staff

      • Special kudos to Suresh Marru, Raminder Singh, David Carver, Stu Martin, Kate Ericson

TeraGrid Quarterly, December 7, 2010

2 5m cpu hours of q3 gateway use source dave hart
2.5M CPU hours of Q3 gateway useSource: Dave Hart

TeraGrid Quarterly, December 7, 2010

Much progress on standardized treatment of community accounts
Much progress on standardized treatment of community accounts

  • Victor Hazlewood, Matthew Woitaszek, Jim Marsteller

  • Nancy’s goal is to provide gateway developers a menu of what to expect at TG sites

  • Fold all paper agreements into single, existing TG user responsibility form

TeraGrid Quarterly, December 7, 2010

Nancy s end of program vision for community accounts
Nancy’s “end of program” vision for community accounts accounts

  • Access to resources via gsi-ssh and GRAM

  • If direct logins to community accounts are restricted:

    • Allow identified developers to “su in”

  • If execution directories are restricted:

    • Provide developer controls through commsh

TeraGrid Quarterly, December 7, 2010

  • Uintah and PET accounts

    • Gateway support staff and documentation are so good, need for advanced support nearly eliminated for large scale CFD gateway Uintah

      • Much to our surprise, Uintah developer presents at TG10 on nearly finished gateway after 20 minute phone conversation with Matthew Woitaszek and pointers to documentation!

      • New work outlined on Population-Environment-Technology (PET) model gateway

TeraGrid Quarterly, December 7, 2010

Analytical ultracentrifugation emerging computational tool for the study of proteins
Analytical Ultracentrifugation accountsEmerging computational tool for the study of proteins

  • Samples from researchers all over the world

    • Some (Germany, Australia) have their own ultracentrifuges and use only the analysis capabilities, others send samples to UT to spin

  • Spin the samples at high speeds, learn about macromolecule properties

  • Monte Carlo simulations

  • Observations are electronically digitized and stored for further mathematical analysis

The Center for Analytical Ultracentrifugation of

Macromolecular Assemblies, UT Health Sciences

TeraGrid Quarterly, December 7, 2010

Source: Suresh Marru, IU

Comprehensive data analysis environment
Comprehensive data analysis environment accounts

  • Management of analytical ultracentrifugation data for single users or entire facilities

  • Support for storage, editing, sharing and analysis of data

    • HPC facilities used for 2-D spectrum analysis and genetic algorithm analysis

      • TeraGrid (~2M CPU hours used)

      • Technische University of Munich

      • Juelich Supercomputing Center

  • Portable graphical user interface

  • MySQL database backend for data management

  • Over 30 active institutions

TeraGrid Quarterly, December 7, 2010

Source: Suresh Marru, IU

Gateway and asta support a growing trend
Gateway accountsand ASTA supporta growing trend

  • TeraGrid advanced support

    • Fault tolerance

    • Workflows

    • Use of multiple TG resources (using Lonestar, expanding to QueenBee and Ranger, using Quarry for test server, waiting for GRAM5 on Ranger)

    • Community account implementation

    • Remote steering

    • Improved UI (no manual specification of CPU time)

    • Applying lessons learned from GridChem, LEAD, incorporating new features into OGCE

      • LEAD is portlet-based, Gridchem is java swing client side app, Ultrascan is php and perl-based gateway, all can use OGCE

      • Big MPI app that forks off many independent runs, improvements here will be tackled by TG's advanced support team

TeraGrid Quarterly, December 7, 2010

Source: Suresh Marru, IU

New advanced support work
New advanced support work accounts

  • Ocean Land Atmosphere Simulation (OLAS) group

    • PI Craig Mattocks, UNC

    • Simulation of flooding and inundation

    • Using TG’s gateway hosting service to set up high availability, real-time data server to ingest live data from NCAR's LDM, OPeNDAP and THREDDS services

    • Gateway work includes workflows to execute OLAS coupled models based on triggers from events in LDM data streams

      • Building off similar work in LEAD

TeraGrid Quarterly, December 7, 2010

Emerging gateway through dataone
Emerging gateway through accountsDataONE

  • Proposed MOU would have DataONE appear to TG as data oriented science gateway and TG appear to DataONE as a set of Member Nodes

  • Combining distributed and diverse data sets to create new scientific insight from new syntheses of data

    • Observational integrated with environmental observational data such as NASA’s MODIS data from the ORNL DAAC to generate predictions of bird species migration patterns

    • TG10 paper by Daniel Fink

TeraGrid Quarterly, December 7, 2010

Gateway software listing
Gateway software listing accounts

  • Populate TeraGrid’s information service with gateway software information

    • A search for computational chemistry packages should turn up both commandline software and packages accessible through a gateway

    • Web services and programmatic generation of package listings too

      • So the RENCI science portal folks don’t have to hand-enter 140 applications!

TeraGrid Quarterly, December 7, 2010

New gateway activities in the extension year
New gateway activities in the extension year accounts

  • Helpdesk support expanded

    • From .2 FTE in PY5 to 1.7 in Extension

      • Helpdesk and Condor support, new GIS communities, SimpleGrid extensions

  • Accounting

    • Improved views for gateways now that we have attributes

  • Community accounts

    • Continued work toward improved standardization

  • Prebuilt VMs with gateway software

    • OGCE, SimpleGrid

  • Online tutorials with CI Tutor and the EOT team

    • OGCE, SimpleGrid

  • More example-based documentation

    • Less talk, more action, short videos, based on user feedback

  • Remote vis for gateways – contract delays

TeraGrid Quarterly, December 7, 2010

Targeted support in the extension all staff available for assignments as new projects come in
Targeted Support in the Extension accountsAll staff available for assignments as new projects come in

  • Cactus

    • Meet the needs of several groups with large TG allocations

  • GridChem, PolarGrid, Ultrascan

    • Scheduling, vis, Matlab processing, processing of centrifuge data for large international project


    • Continuing work to combine capabilities

  • SNS


  • OpenSocial for gateways

  • Condor and cloud support

TeraGrid Quarterly, December 7, 2010

TG to XD transitions accountsfor gateways

  • Stu, Nancy and XD gateway leads to conduct focused discussions with each gateway, pending architecture definition

    • Update use case description

    • Add/subtract gateways as necessary

    • Ask about impact of transition (depends on the architecture)

    • Ask about entries in software catalog

    • Ask about attribute-based authentication (also depends on architecture)

  • Assessment and transition of gateway advanced support projects too

TeraGrid Quarterly, December 7, 2010

Gateway sustainability study small non tg eager grant
Gateway Sustainability Study accountsSmall, non-TG, EAGER grant

  • Characteristics of short funding cycles

    • Build exciting prototypes with input from scientists

    • Work with early adopters to extend capabilities

    • Tools are publicized, more scientists interested

    • Funding ends

    • Scientists who invested their time to use new tools are disillusioned

      • Less likely to try something new again

    • Start again on new short-term project

  • Need to break this cycle

  • EAGER grant to look at characteristics of successful gateways and domain areas where a gateway could have a big impact

  • 4 focus group meetings over 2 years

    • First 2 held June, 2010


TeraGrid Quarterly, December 7, 2010

Questions? accounts

TeraGrid Quarterly, December 7, 2010