GIG Software Integration: Area Overview

1. GIG Software Integration:Area Overview TeraGrid Annual Project Review April, 2008

2. Software Integration (SI) Activities • GIG Area Coordination (.73 FTE) • GIG SI Operations (~2.5 FTE) • Package CTSS software (2) • Maintain (own) six CTSS capability kit definitions (.45) • Operate infrastructure services (.1) • GIG SI Development (~2 FTE) • Information Services (1) • TeraGrid Client Software Distribution (.35) • Application Hosting Prototype (.25) • Data and Usage Reporting (.16) • Resource Providers (RP) • CTSS Deployment & Maintenance • RP Capability Development

3. Software Integration (SI) GIG Partners Contributing Institutions: • NCSA (Packaging and Operations) • TACC (MyCluster deployment) • U of Chicago/ANL (most areas) • U of Southern California/ISI (Information Services) • U of Wisconsin (Build & Test, TeraGrid client) Note: All RPs deploy and maintain CTSS. Some RPs are involved in capability development.

4. Operations - Kits Maintain (own) six CTSS 4 capability kits: • Create Capability Kit Definition document • High-level capability description • Documents user focused use scenarios • Create Capability Kit Implementation document • Identifies components and deployment details • Work on software component issues • SI Owned Kits • TeraGrid Core Integration • Remote Login • Remote Computation • Application Development & Runtime Support • Science Workflow Support • Parallel Application Support

5. Operations – Package CTSS • Build binaries and package 11 CTSS components • Coordinated 9 with OSG’s VDT • Package 24 other CTSS components • In response to: • Security vulnerabilities • Software issues • RP technical requirements • New resources and platforms (NCAR, NCSA Abe, LONI/LSU) • New operational requirements (metrics) • Highlights: • Capability registration packages • Several GridFTP upgrades • Several Information Services deployments • HPN-SSH • Usage Logging • GRAM Audit Beta (production in 2008)

6. Operations – Operate Services Operate Infrastructure Services: • Build & Test central server • TeraGrid wide information services servers (2) • Primary and hot stand-by servers • Multiple user facing services • Aggregates and caches RP information services • High availability • GIG owned development/testing servers

7. Operations – Challenges • Addressing scalability and reliability issues • Brief LEAD activity summary • Brief MPICH-G2 activity summary • Complexity in new capabilities • Brief description of Remote Compute version 4 effort • Other challenges mentioned in AR

8. Motivation for CTSS 4 Paradigm Shift • Increasing diversity in systems had resulted in the CTSS definition being riddled with exceptions • The stack model didn’t document user scenarios, making it: • Hard for users to understand what they could do with CTSS • Hard to know what impact changes or upgrade to components would have • Difficult to correlate usage data for individual components to specific user benefits or user types • Difficult/impossible to evaluate possible competitors to CTSS components (or proposals to eliminate components), • Difficult to make judgments about where to devote effort in regard to CTSS (for GIG, RPs, and users) • Hard to have objective discussions about the cost/benefit of specific components on specific RP systems

9. Significant Paradigm Shift • Paradigm shift for how we describe CTSS • From CTSS 1 thru 3 monolithic software stack • To CTSS 4 modular and optional capabilities • Improved many aspect of capability delivery • Better coordinated capability description (to users and internally) • Improved capability documentation organization • Improved capability availability information • More focused package, deploy, and configure process • How RPs choose and publish what they offer • Timeline • Designed in 2006 • Capability kits defined in Q1-2 2007 • Capabilities rolled out in Q2-Q3 2007

10. TeraGrid RP Kit Support

11. Development – Information Services • Importance • Enables the TeraGrid’s “Common User Environment” • SI Activities • Design schemas used to publish information • Develop information providers that gather and publish information • Develop information views • Develop new information access • Design and implement information aggregation and persistent caching

12. Development – Projects • TeraGrid client software distribution • Design and package that organizations outside the TeraGrid can deploy to enable users to access TeraGrid resources • Application hosting capability • Prototype the ability for TeraGrid partners or users to dynamically provision application and services on TeraGrid resources • Usage reporting • Effort to instrument TeraGrid grid services (primarily GridFTP) with extended metrics and usage reporting capability • Build & Test • Tool that manages software build “jobs” and tracks inputs, build options, and build results • Benefits • Where does it come from • How are we using it • Collaborations with other global grids in the areas of: • Common Grid publishing schemas • Common execution environment standards • Capability testing best practices • Participated in OGF sponsored Grid Interoperability Now “GIN” activity and testbed

13. Area Coordination • Define and own CTSS definition process • Used by multiple project areas • Define and own CTSS change management process • Used by multiple project areas • Coordinate CTSS deployment • Facilitate Software working group meetings • Participate in Operations and other CTSS support working groups • Participate in International Standards and Outreach • Generic area coordination • Establish area goals and planning in coordination with other areas • Budget, staffing, and sub-Awards • Reporting and reviews