Future Work

1. Future Work

2. Project Timeline: Evolving ESG for the Future

3. Putting the Next-Generation ESG into Production Milestone June 2009: ESG-NCAR & ESG-PCMDI deployed as beta, open to users for beta testing Milestone July 2009: ESG-NCAR & ESG-PCMDI deployed as version 1, operational status Add ORNL data node Republish C-LAMP data through new ORNL data node Shut down standalone C-LAMP ESG portal Add LANL, NERSC data nodes

4. On-Going Support for CMIP5 and IPCC AR5 • CMIP5 coordinated climate experiments: • Modeling groups run experiments 2009-2010 • Model data archived, analyzed and assessed 2011-2012 • AR5 WG1 assessment completed 2013 • Further CMIP5 model experiments (not assessed in AR5) run and analyzed 2011-2013 • ESG-CET coordinated software efforts: • Deal with system integration issues and further development of production system 2009 • Continue infrastructure development (e.g., local and remote analysis) 2009 - 2011 • Support progressive deployment of Data Nodes 2009-2011 • Support AR5 WG1, WG2, and WG3 2009-2011

5. CMIP5 International Testbed • Establishing initial testbed environment for international federated ESG • US: PCMDI, NCAR, ORNL, GFDL • UK: BADC • Germany: MPIM • Japan: JAMSTEC and U. Tokyo (joint) • Start international testbed after cut-over of production ESG to next-gen software, July 2009 • Test behavior of federation, bulk data movement, replication over very wide area • Test software packaging and deployment • Publish and federate CMIP3 (IPCC AR4) data

6. Interoperability non-ESG Software Stacks • Some sites (e.g., BADC, DKRZ) have their own climate data management systems • Prefer not to maintain separate ESG installation (in the long run) • Data services for IPCC WG2 and WG3 communities (BADC, DKRZ responsible) probably require deeper integration • Formalize requirements and APIs for service-level interoperability • SAML-based user attributes service • SAML-based authorization service • Support authorized direct OpenDAP access • OAI-PMH for metadata exchange

7. Near-Term ESG Development Plans Global services Registry service Monitoring services Full implementation of data versioning (publishing, search, access) Support direct files download via DML UI for data sub-setting requests to TDS Ingestion of metrics from data nodes Use SAML for exchanging user attributes, authorization statements with federated Data Centers Secure data access via LAS Review and upgrade all UI elements, workflows Cross-browser testing Packaging & Documentation Scalability, performance and stress testing

8. Longer-Term ESG Development Plans • Support direct discovery and download of large collection of files (w/ bulk data movement client) • Aggregation of metrics across Gateways • Research strategies to facilitate search & discovery from Google and other search engines • Integrate GIS services for display and data integration • Research integration with Google Earth • Customization/branding capabilities for Gateway • Develop user workspace functionality • Develop web services for data query and access • Support scientific workflows

9. Enabling Rich Client Access to ESG Fourth generation scripting languages are extremely popular in climate community for analysis and visualization (e.g., CDAT, NCL, Ferret) Server-side integration of these tools is valuable, but limited to pre-programmed products Also want to give these tools ability to interact directly with ESG to allow fully scripted client-side usage No need for separate data staging step

10. Workflow: Streamlining Data Exploration and Visualization Server-side data analysis could become quite complex Multiple tools, used in proper sequence Computationally intensive Data at multiple sites, coordination of movement Time delays from deep storage retrival, queuing for CPU resources Use workflow approaches to help orchestrate Possible tools include Kepler, Tarverna, and Swift

11. ESnet: Scaling the Earth System Grid to 100 Gbps • Scaling the Earth System Grid’s network capacity to meet the high volumes data requests by high volumes of users. • Integrating perfSONAR and NetLogger into the ESG’s administration tool suite

12. Scaling the ESG Hardware Infrastructure LLNL to upgrade storage capacity to 1 PB over the next 18 months to accommodate CMIP5 “common core” archive. Also upgrading its compute server that will handle data requests ORNL’s National Center for Computational Sciences (NCCS) is committed to support CMIP5’s needs for computing and data dissemination, as well as other climate-related uses of the Center (e.g., CCES, C-LAMP) NCAR recently purchased a 7-server cluster to support more intensive ESG operations Cluster deployment allows adding hardware if needed

13. Support and Maintenance of the Production ESG Help desk Assistance for users – already providing, but expect larger user base, possibly with lower levels of experience and expertise (e.g., IPCC WG2 and WG3 communities) Assistance for site operators – new, due to broader deployment Long-term support Many opportunities for further development of ESG… But even if developmentstopped at the end of ESG-CET, climate community still depends (increasingly) on the ESG service Issue needs consideration IPCC deadline for paper submissions End of ESG2 SciDAC Project IPCC AR4 released Interest continues andgrows!

14. Scientists From remote sites, scientists search a climate portal containing petabytes of high-resolution regional central Africa data… Unproductive – so scientists run several models in real-time to generate ensemble simulations of African climate … Using server-side visualization tools, they are able to simultaneously view and annotate plots of ensemble climate statistics Vision for the Future (Part 1)

15. Vision for the Future (Part 2) Policy Maker Later a malaria policymaker discovers (using a “new search capability”) the provenance of the scientists saved session… Policymaker conducts further assessment and re-analysis of the derived datasets before reducing to 20 terabytes and moving to local workstation for further study