Ralph Roskies Scientific Director Pittsburgh Supercomputing Center

1. PSC RP highlightsTeraGrid QuarterlySept 6, 2007 Ralph RoskiesScientific DirectorPittsburgh Supercomputing Center

2. Science Impact

3. Unprecedented Storm Forecast Experiments • 2007 NOAA and University of Oklahoma Hazardous Weather Testbed (HWT)Spring Experiment[reservations, real-time forecasting, performance improvement] • Major goal: assess how well ensemble forecasting works to predict thunderstorms, including the supercells that spawn tornados. • First time ensemble forecasts are being carried out at high enough spatial resolution. • First time ensemble forecasts are being carried out in real time in an operational forecast environment. • Requires >100× more computing daily than the most sophisticated National Weather Service operational forecasts. • “This is unique – both in terms of the forecast methodology and the enormous amount of computing. The technological logistics to make this happen are nothing short of amazing.”– Steven Weiss, Science and Operations Officer of the NOAA Storm Prediction Center (SPC) • “This experiment represents an enormous leap forward. Ensembles open up a new array of interpretative capabilities to forecasters analyzing how good the forecast is. With ensembles, you're not only forecasting the weather, you're forecasting the accuracy of the forecast.”—Kelvin Droegemeier, LEAD Director, University of Oklahoma. • Collaborators: NOAA National Severe Storms Laboratory, Center for Analysis and Prediction of Storms (CAPS), LEAD (Linked Environments for Atmospheric Discovery), SPC, PSC, NCSA From Quarks to the Cosmos: Enabling Computational Science at PSC · August 17, 2007 3

4. Enabling CAPS Daily Forecasts at PSC • PSC optimized WRF for the Cray XT3, gaining a 3× speedup in I/O, substantially improving overall performance. PSC also optimized the I/O forpost-processing routines used to visualize and analyze the forecast output, achieving 100× speedup. • PSC modified the reservation and job-processing logic of its job-scheduling software to implement auto-scheduling of WRF runs and related post-processing, 760 separate jobs each day, demonstrating the TeraGrid's ability to use the Cray XT3, a very large capability resource, on a scheduled, real-time basis. • PSC networking staff coordinated with OneNet, a regional network of the State of Oklahoma, and National Lambda Rail (NLR), a network initiative of U.S. universities, and with Cisco Systems, who contributed use of a dedicated lambda for up to a 12-month period, allowing the transfer of 2.6 TB of data per forecast day.

5. Providing 3D Visualization to the Ocean Community • Zulema Garraffo et al. (RSMAS, Univ. of Miami) asked to visualize her group’s increasingly large data close to where it is generated. • Although flow in our oceans is inherently 3D, HYCOM vis has relied principally on 2D contour plots. • Building on PSC’s 13-year collaboration with the ocean modeling community, PSC scientists John Urbanic and Kent Eschenberg collaborated closely with Garraffo, George Halliwell (also RSMAS), and Alan Wallcraft (NRL) to develop 3D vis capability for HYCOM results. • Also improved file management, scheduling, post-processing scripts

6. PSC’s Ocean Film Showing at Carnegie Museum • Path of the Titanic Across the Gulf Stream:Film showing HYCOM simulation of oceancurrents along the Titanic’s actual route • Showing hourly in the Carnegie Museumof Natural History’s Earth Theater,together with the “Night of the Titanic” exhibit • http://www.carnegiemnh.org/exhibits/etschedule.html • Research: Zulema Garraffo, George Halliwell (Univ. of Miami),and Eric Chassignet (Florida State University)

7. Membrane Remodeling by N-BAR Domains Endocytosis:the life-sustaining process by which cells absorb material from outside the cell by bending their membrane to form a “vesicle” and engulfing it. • Banana-shaped proteins known as BAR domains BAR domains change liposomes to tubules (tubulate) in vitro… • Working approach combines large-scale MD simulation, every applicable piece of atomistic-level information, and an elastic membrane model to achieve multiscale modeling. • “The XT3 has been amazing. We haven’t found a hard limit on scaling up the number of processors.” Lee and Schekman,Science 303, 479 (2004). Lodish et al., Molecular Cell Biology, 4th ed. Image courtesy Greg Voth Mesoscopic image of a charged phospholipid vesicle undergoing the beginnings of BAR domain induced tubule formation. Gregory Voth(Univ. of Utah) andPhilip Blood (PSC)

8. Molecular Dynamics N-BAR Simulations using NAMD • BAR domain structure from Protein Data Bank (1URU); CHARMM27 force field • 738,000 atoms (50×10 nm bilayer) • NAMD (>2ns/day running 738,000 atom system on 512 processors) • “Close collaboration with PSC consultants to optimize calculations for the XT3, significantly reducing time to solution” Blood, P. D. and Voth, G. A. Proc. Natl. Acad. Sci. 103, 15068 (2006)

9. Enabling Breakthrough NAMD Simulations at PSC • Problem: In NAMD2, PE0 holds a persistent PDB data structure and a map of the distributed data structures and the Cartesian coordinates and atom types for entire molecular system. • Its memory requirement increases with problem size, prohibiting certain simulations of interest on a system with only 1GB of memory per node. • Solution: 2 nodes on bigben are now configured with 4GB of memory and accessed through the phat queue. • The phat queue is being used for production calculations by the Schulten and Voth groups to study dynamics of ribosomes and N-BAR domains, respectively. • 3M atom ribosome calculations sustain >2.08 Tflop/s on 2048 processors. • PSC is (still?) the only XT3 site that allows users to assign particular logical nodes to specific physical nodes. • PSC (Shawn Brown) has also collaborated with the Kalé group to correct flow control issues and to optimize NAMD.These improvements are now part of the public NAMD distribution, benefiting all NAMD users.

10. Workshops

11. Petascale Applications Symposium:Multilevel Parallelism and Locality-Aware Algorithms • June 22-23, 2007 at PSC • Addressed omnipresence of multicore processors,especially the assemblage of large numbers of tightly coupled, multicore processors into leadership-class supercomputers • Invited speakers included developers of algorithms, languages, and strategies that cross-cut application domains, to stimulate cross-fertilization of ideas and formation of new collaborations. Extensive discussion and a panel engaged all participants. • 53 participants (including 6 women and 1 undergraduate) represented 24 institutions • Synopsis, presentations, and video available at http://www.psc.edu/seminars/PAS/proceedings.php Sanjay Kalé discussing Petascale andMulticore Programming Models: What Is Needed in Saturday morning’s session

12. Toward Multicore Petascale Applications:Broadly Applicable Training for the TeraGrid (Aug 27-29) • Goal: Train developers to optimize applications for multicore processors • Audience: Scientists and engineers who would like their codes to run efficiently on modern architectures, including both current and prospective users of the TeraGrid • Applicability: Workshop presented on the XT3; ideas and techniques applicable to multicore processors and systems across the TeraGrid • 19 participants including 4 women, 9 institutions

13. Adding Value:Performance Engineering

14. Boosting Cray SeaStar Performance at PSC • PSC’s enhancements to Cray’s job placement algorithm reduce contention by optimizing compactness and minimizing fragmentation. • Improves performance of communication-intensive applications such as NAMD and DNS by up to 10% under realistic workloads. • PSC applied hardware performance counters to NSF workloads to optimize the packet routing policy and packet aging clock tick in the SeaStar router. • Reduces average message latency by 14% and significantly boosts observed performance. • Enabling 4 virtual channels reduces high variance in packet latency. • Dramatically improves communication-intensive operations. Resulted in new firmware appearing in Cray’s distribution. • These optimizations have been presented, published, and made available to other sites through new releases of Cray firmware and software.

15. PSC leads GIG efforts in • Data Movement and Speedpage • Lustre WAN • Security Operations • User Support Coordination • Science Writing • Evolution of accounting to meet future needs

16. End-to-End Simulation: A Petascale Paradigm • End-to-end applications being developed through user/PSC collaborations feed directly into end-to-end data work PSC is doing for the GIG. • Cosmology (Thomas Quinn, Univ. of Washington; Jeff Gardner, PSC) • Derek Simmel (PSC) is using this workflow to understand and fix GridFTP transfers in conjunction with computations on BigBen (PSC) and Cobalt (NCSA). (Derek has also developed significant insights into data staging methods with MyCluster/Gridshell, including WAN filesystem use cases.) • Molecular dynamics (Gregory Voth, Univ. of Utah; Phil Blood, PSC) • Phil Blood has described in detail the data and work flow he used when he was part of the Voth group. This info is currently being processed by the Data team. • Seismic simulation (Jacobo Bielak and David O’Hallaron, CMU; Raghu Reddy, PSC) • Raghu Reddy, in his capacity as GIG user services coordinator, is helping SCEC with their work and data flow. As a PSC consultant, he has also helped the Toomre group solve TG data transfer problems. Raghu’s insights are also being fed to the Data team. • Storm Forecasting (Ming Xue, Univ. of Oklahoma, Norman; David O’Neal, PSC) • Outstanding CAPS 2007 Spring Campaign: first time ensemble forecasts are being carried out in real time in an operational forecast environment.

17. Other TeraGrid leadership roles • Sergiu Sanielevici leads Extreme Scalability RAT • Laura McGinnis co-leads the RAT on HPC University, and Sergiu is co-PI of the TeraGrid CI-Team proposal, "Pathways to Broadening Participation in TeraGrid

18. Other recent achievements for TeraGrid • Aaron Shelmire • did significant work on securing community accounts • Ben Bennett- • plugged security hole in GX-Map and in Grid-proxy-init in Globus Toolkit • Deployed a TG version of PSC's NPAD/pathdiag "one click" diagnostic service on TeraGrid network monitoring (netmon) machines. • provides "trivial" diagnostics of "last mile" and end-system problems, specifically the short path from the netmon system to a local host and of the local host itself. • The TeraGrid Data Working Group recommended that all TeraGrid sites deploy servers with PSC’s (Chris Rapier’s) hpn-ssh for users. • Mounted GPFS on XT3 test system

19. Other activities • TG07 • 3 talks • 3 tutorials • 2 student contest winners, result of project JumpStart, led by Laura McGinnis • Prepared and delivered material to introduce high school students to concepts of cyberinfrastructure and the TeraGrid. • Material presented to our K-12 outreach workshop participants (teachers) and at two sessions for local high school students and teachers. • 15+ teachers and 8 students attended the sessions, resulting in two submissions to the TeraGrid Student Conference.