Testing and Validation of the NPACKage

1. Testing and Validation of the NPACKage NPACI All-Hands Meeting March 19, 2003 Marty Humphrey Assistant Professor Computer Science Department University of Virginia

2. NPACKage Project Structure • Participation across the partnership • UCSB, SDSC, UVA, ISI • Dedicated staff at each site • Software contributors • Activities focused under current NPACI allocation • NPACKage specific activities • Packaging team • Builds team • Testing team • Resource leads • Responsible for local deployment • User Services • Executive Committee • Frequent review of progress

3. What’s So Tough about Software Test? • Software has lousy quality historically • Simple program can have huge space of execution • “Testing is not innovation but rather verification”

4. Why Does Software Have Bugs? • miscommunication • software complexity • programming errors • changing requirements • time pressures • poorly documented code • bugs in software development tools

5. Basics of Software Testing • Unsatisfied goals • Find cases where the program does not do what it is supposed to do • Unwanted side effects • Find cases where the program does things it is not supposed to do

6. Methodology • Define the expected output or result. • Don't test your own programs. • Inspect the results of each test completely. • Include test cases for invalid or unexpected conditions. • Test the program to see if it does what it is not supposed to do as well as what it is supposed to do. • Avoid disposable test cases unless the program itself is disposable. • Do not plan tests assuming that no errors will be found. • The probability of locating more errors in any one module is directly proportional to the number of errors already found in that module.

7. Terminology • White box testing • Based on knowledge of internal logic/source code • Black box testing • Not white box testing :^) • Unit testing • To test particular functions or code modules • Integration testing • Testing of combined parts

8. Terminology (cont.) • Performance / Stress testing • Good for testing scalability • Regression testing • Re-testing after modifications • Acceptance testing • Performed by end-users • Completion criteria • How do you know when you’re done?

9. Don’t the individual projects test their software? • Yes, of course, but we can do more • Value of centralized testing • Complex interaction tests can be better achieved organizationally, not in isolation • Incomplete testing of any component reflects poorly on the organization (NPACI) • Independent testing can greatly improve software quality

10. Value Added at UVa • Work closely with other NPACKagers • NPACKage packagers (Larry Miller, UCSB; Bill Link, SDSC) • NPACKage “Builders” and “Testbed Deployers” (Mats Rynge, USC/ISI) • UVa: RH Linux 7.3 (8.0?), AIX 4.3, [Solaris] • Combination of “black box” and “white box” • Use Drivers and Stubs provided by NPACKage contributors • Provide feedback to NPACKage contributors • Goal: Testing Deployment • “Given a Deployment, does it work?” • NPACI Testbed and NPACI Production • Goal: Testing Deployability • Broader impact in community

12. Software for the NPACKage • NMI • Globus, Condor-G, NWS, GSI-OpenSSH, KX.509, GridConfig, (MyProxy, MPICH-G2) • DataCutter • Storage Resource Broker – SRB • GridPort • Ganglia • APST • GridSolve • LAPACK For Clusters – LFC

13. Testing of NMI: SURA • UVa: part of SURA NMI Testbed • Globus • Condor-G • NWS • GSI-OpenSSH • UVa co-PI on MyProxy NMI grant

14. Testing of DataCutter • Last extracted from CVS on March 11, 2003 • Tests from developers a. Make sure the daemons start g. Placement test b. Buffer test h. Layout test c. Instance test i. Cluster test d. Returns test j. Directory test e. Crash test k. Grid array averager f. Endian test • Thanks to Shannon Hastings

15. Testing of SRB • Just received SRB 2.0.0/2.0.1 early this week • Plan • Exercise unit tests in SRB (awaiting descriptions) • Interactions with other NPACKage components • DataCutter • GridPort • GridFTP • GSI • Many clients, server • Thanks to Reagan Moore, Wayne Schroeder

16. Testing of GridPort • Last extracted (v 2.2) from CVS on March 14, 2003 • Run test Perl scripts from distribution • We use in our Alpha Project • “Protein Folding on the Grid” (with C. Brooks, M. Crowley of TSRI) • Thanks to Maytal Dahan, Mary Thomas

17. Testing of Ganglia • Last extracted from CVS on March 11, 2003 • Currently executing on 9 nodes on UVa’s Centurion • Tests • Start a ‘gmond’ and pull its XML tree (“telnet localhost 8649”); verify with “xmllint --verify out.xml“ • Planned • The ganglia-python package from SDSC's Rocks group includes an MDS provider that obtains data from ganglia. • Is the information correct? • Thanks to Federico Sacerdoti, Phil Papadopoulos

18. NPackage Testbed • Machine contributions from resource sites • Many thanks for the access to the systems for development and testing, especially: • U.Mich • 6 AIX4.3, 1 ia64 RH7.1 • SDSC • Cluster: 4 RH7.3, 2 AIX5.1 • UCSB • 1 SUSE 8.1 • USC/ISI • 1 RH7.2 • Point your favorite LDAP browser at: • giis.npaci.edu:2135

19. Testing NPACKage Stage-1 • Resource discovery and monitoring infrastructure • Hierarchical discovery and monitoring cache/index (MDS) • Host resource information (MDS) • Cluster resource inform (Ganglia) • Binaries • ia64-redhat-7.1 • power3-aix-4.3 (testing...) • power3-aix-5L • x86-redhat-7.2 • x86-redhat-7.3 (tested) • x86-suse-8.1 • Is the information correct?

20. Automation • Some tools exist • Dart • Tinderbox / TreeFire (Mats) • “Inca Build System” (TeraGrid) • Issue: Determining Cause/Blame • Builder • Deployer • Packager • Resource Provider • Software Developer

21. Bottom Line • NPACI software infrastructure is transitioning to production • Evolving plan for testing and validation Providers’ tests  tests at UVa  tests on NPACI testbed  deployment to NPACI sites • Integrate and influence NMI testing and TeraGrid testing • Next: APST v2.0, GridSolve, LFC • NPACKage: infrastructure (and soon) integration with Applications (e.g., CHARMM Portal)