1 / 59

Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl

The U.S. DOE Advanced CompuTational Software (ACTS) Collection. Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl.gov. OUTLINE. Motivation Introduction to the DOE ACTS Collection Interfaces to the ACTS Collection Software Sustainability Requirements References.

dior
Download Presentation

Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. The U.S. DOE Advanced CompuTational Software (ACTS) Collection Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl.gov SIMULA Research Laboratory - May 2008

  2. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

  3. http://acts.nersc.gov/MatApps Where are the applications? Development of High End Computer Simulations • Accelerator Science • Astrophysics • Biology • Chemistry • Earth Sciences • Materials Science • Nanoscience • Plasma Science • : • Commonalities: • Major advancements in Science • Increasing demands for computational power • Rely on available computational systems, • languages, and software tools SIMULA Research Laboratory - May 2008

  4. min[time_to_solution] (production) (Software Evolution) (Long-term deliverables) min[software-development-cost] max[software_life] and max[resource_utilization] Software Development and Evolution min[time_to_first_solution] (prototype) • Outlive Complexity • Increasingly sophisticated models • Model coupling • Interdisciplinary • Sustained Performance • Increasingly complex algorithms • Increasingly diverse architectures • Increasingly demanding applications SIMULA Research Laboratory - May 2008

  5. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

  6. THE U.S. DOE ACTS COLLECTION Goal: The Advanced CompuTational Software Collection (ACTS) makes reliable and efficient software tools more widely used, and more effective in solving the nation’s engineering and scientific problems. • References: • L.A. Drummond, O. Marques: An Overview of the Advanced CompuTational Software (ACTS) Collection. ACM Transactions on Mathematical Software Vol. 31 pp. 282-301, 2005 • http://acts.nersc.gov SIMULA Research Laboratory - May 2008

  7. The Advanced CompuTational Software Collection (ACTS) • Components: • Solid Base: non-commercial and open source tools developed at DOE laboratories and universities. • Independent Tool Evaluations and Consultation provided through acts-support@nersc.gov • High Level User Support problem identification, tool and interface selection, specific tuning parameter configurations, installation, documentation, etc. • Training and Dissemination workshops, lectures, active conference participation (acts.nersc.gov. • Collaborations with HPC centers, computational sciences research centers (national and international level), and software and computer vendors. SIMULA Research Laboratory - May 2008

  8. SIMULA Research Laboratory - May 2008

  9. Changes in algorithms sometimes lead to several years advancement in computations. Needs Flexibility! Its performance is influenced by system parameters and in steps in the algorithm. Critical points: portability and scalability. New Architecture requires extensive tuning, may even require new programming paradigms. This is Difficult to maintain and not “very” portable. Software Sustainability Algorithmic Implementations I/O Application Data Layout Control Tuned and machine Dependent modules SIMULA Research Laboratory - May 2008

  10. Software Sustainability USER's APPLICATION CODE (Main Control) Compilers + Expert Drivers + Support AVAILABLE AVAILABLE Algorithmic Implementations AVAILABLE I/O Application Data Layout LIBRARIES LIBRARIES & PACKAGES LIBRARIES & PACKAGES Tuned and machine Dependent modules SIMULA Research Laboratory - May 2008

  11. Scientific or engineering context • Domain expertise Critical Path for HPC Software Stack • Simulation codes • Data Analysis codes General Purpose Libraries • Algorithms • Data Structures • Code Optimization • Programming Languages • O/S - Compilers Hardware - Middleware - Firmware SIMULA Research Laboratory - May 2008

  12. Funded by DOE/ASCR Library Development Numerical Tools Code Development Run Time Support http://acts.nersc.gov • Algorithms • Data Structures • Code Optimization • Programming Languages • O/S - Compilers Critical Path for HPC Software Stack General Purpose Libraries Hardware - Middleware - Firmware SIMULA Research Laboratory - May 2008

  13. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  14. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  15. Structure of PETSc PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK SIMULA Research Laboratory - May 2008

  16. Linear System Interfaces Linear Solvers GMG, ... FAC, ... Hybrid, ... AMGe, ... ILU, ... Data Layout structured composite block-struc unstruc CSR Hypre Conceptual Interfaces SIMULA Research Laboratory - May 2008

  17. Hypre Conceptual Interfaces to Solvers List of Solvers and Preconditioners per Conceptual Interface SIMULA Research Laboratory - May 2008

  18. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  19. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  20. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  21. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  22. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  23. TAO - Interface with PETSc SIMULA Research Laboratory - May 2008

  24. OPT++ Interfaces • Four major classes of problems available • NLF0(ndim, fcn, init_fcn, constraint) • Basic nonlinear function, no derivative information available • NLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information available • FDNLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information approximated • NLF2(ndim, fcn, init_fcn, constraint) • Nonlinear function, first and second derivative information available SIMULA Research Laboratory - May 2008

  25. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  26. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

  27. ACTS Tools: Functionality SIMULA Research Laboratory - May 2008

  28. ACTS Tools: Functionality SIMULA Research Laboratory - May 2008

  29. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

  30. How Does One Use ACTS Tools? Linear System Interfaces • -ksp_type [cg,gmres,bcgs,tfqmr,…] • -pc_type [lu,ilu,jacobi,sor,asm,…] More advanced: • -ksp_max_it <max_iters> • -ksp_gmres_restart <restart> • -pc_asm_overlap <overlap> • -pc_asm_type <. . > Linear Solvers GMG FAC Hybrid, ... AMGe ILU, ... Data Layout structured composite blockstrc unstruc CSR CALL BLACS_GET( -1, 0, ICTXT ) CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL ) : CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL ) : : CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB, $ INFO ) Language Calls Command lines Problem Domain SIMULA Research Laboratory - May 2008

  31. TAU PETSc Ex 2 Ex 1 Tool to Tool Interoperability One Side Interoperability TOOL B TOOL C TOOL A TOOL F TOOL E TOOL D SIMULA Research Laboratory - May 2008

  32. PyACTS matlabMPI Star-P NetSolve High Level Interfaces High-level User Interfaces to the ACTS Collection Ax = b View_field(T1) User OPT++ PAWS CUMULVS TAU Globus Hypre Global Arrays Chombo AZTEC PETSc PVODE Overture ScaLAPACK SuperLU TAO SIMULA Research Laboratory - May 2008

  33. PyACTS Vicente Galiano Miguel Hernandez University Tony Drummond Lawrence Berkeley National Laboratory Violeta Migallón and José Penadés University of Alicante Goal: Provide a didactical tool to the ACTS collection. Provide a Python based interface to the ACTS Collection. • References: • L. A. Drummond, V. Galiano, O. Marques, V. Migallon, J.Penades: PyACTS: A High-level Framework for Fast Development of High Performance Applications. Lecture Notes in Computer Sciences, Vol. 4395, pp 417-425, 2007. SIMULA Research Laboratory - May 2008

  34. PyACTS PyACTS PyScaLAPACK PySuperLU PyACTS Wrappers ScaLAPACK Wrappers SuperLU Wrappers Python World PyMPI NumPy . . . ScaLAPACK SuperLU Python SIMULA Research Laboratory - May 2008

  35. PyACTS: Basic Services • BASIC Services: Creation and modification of different data objects and parallel environment specifications (matrices, data layouts, ctx,) • I/O Services : Parallel read/write. Currently supported ASCII and NetCDF. • Verification and Validation: Predicates and parameter type checking. • Data Conversion. Interoperable objects between libraries. SIMULA Research Laboratory - May 2008

  36. PyACTS: Motivation PyClimate (J. Saenz et al,Univ. Basque Country) • Support to common tasks during the analysis of climate variability data. • Simple IO operations • Operations with COARDS-compliant NetCDF files • Empirical Orthogonal Function (EOF) analysis, • Canonical Correlation Analysis (CCA) • Singular Value Decomposition (SVD) analysis of coupled datasets • Some linear digital filters • Kernel based probability-density function estimation and • access to DCDFLIB.C library from Python. SIMULA Research Laboratory - May 2008

  37. PyACTS: Performance in PyClimate EOF calculations Empirical Orthogonal Function (Day calc) SIMULA Research Laboratory - May 2008

  38. PyScaLAPACK: pvgesvd Performance SIMULA Research Laboratory - May 2008

  39. PyACTS: Performance > from PyACTS import * > import PyACTS.PyPBLAS as PyPBLAS > import time > n=500 > ACTS_lib=1 # ScaLAPACK library > PyACTS.gridinit() # grid initialization > alpha=Scal2PyACTS(2,ACTS_lib) # convert scalar # to PyACTS scalar > beta=Scal2PyACTS(3,ACTS_lib) > a=Rand2PyACTS(n,n,ACTS_lib) # generate a random # PyACTS array > b=Rand2PyACTS(n,n,ACTS_lib) > c=Rand2PyACTS(n,n,ACTS_lib) > c=PyPBLAS.pvgemm(alpha,a,b,beta,c) # call level 3 # PBLAS routine > PyACTS.gridexit() c=PyPBLAS.pvgemm(alpha,a,b,beta,c) SIMULA Research Laboratory - May 2008

  40. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

  41. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

  42. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

  43. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

  44. Versions (tools, systems, O/S, compilers) • Sanity-check (robustness) • Interoperability (maintained) • Consistent Documentation Software Quality Assurance • Robustness • Scalability • Extensibility • Interoperability • User Friendliness • Documentation • Periodic test and evaluations • (test engines and dependency graphs) SIMULA Research Laboratory - May 2008

  45. ScaLAPACK PBLAS Global Local LAPACK BLACS platform specific BLAS MPI/PVM/... ScaLAPACK’s Software Structure SIMULA Research Laboratory - May 2008

  46. + * 2.2 GHz AMD Opteron * * + BLAS: Basic Linear Algebra Subroutines • Level 1 BLAS: vector-vector • Level 2 BLAS: matrix-vector • Level 3 BLAS: matrix-matrix BLAS LEVELS: • Design Considerations: • Portability • Performance: development of blocked algorithms is important for performance! SIMULA Research Laboratory - May 2008

  47. ScaLAPACK: Data Layouts • 1D block and column distributions • 1D block-cycle column and 2D block-cyclic distribution • 2D block-cyclic distribution used in ScaLAPACK for dense matrices SIMULA Research Laboratory - May 2008

  48. Astrophysics Applications Cosmic Microwave Background Analysis, BOOMERanG collaboration, MADCAP code (Apr. 27, 2000). • The statistics of the tiny variations in the CMB (the faint echo of the Big Bang) allows the determination of the fundamental parameters of cosmology to the percent level or better. • MADCAP (Microwave Anisotropy Dataset Computational Analysis Package) • Makes maps from observations of the CMB and then calculates their angular power spectra. (See http://crd.lbl.gov/~borrill). • Calculations are dominated by the solution of linear systems of the form M=A-1B for dense nxn matrices A and B scaling as O(n3) in flops. MADCAP uses ScaLAPACK for those calculations. SIMULA Research Laboratory - May 2008

  49. PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK PETSc Image Provided by PETSc Development Team, ANL) SIMULA Research Laboratory - May 2008

  50. Synchronization Points Basic Conjugate Gradient Algorithm Scalars , , y  Vectors x, r, p (= search direction), and q SIMULA Research Laboratory - May 2008

More Related