Synthetic Grid Workloads with Ibis, K OALA , and GrenchMark

1. Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark A. Iosup, D.H.J. Epema Jason Maassen, Rob van Nieuwpoort PDS Group, EEMCS, TU Delft Computer Science Dept., VU Amsterdam CoreGRID Integration Workshop, Pisa

2. Collaboration context • The Partners • WP6Technische Universiteit Delft (TU Delft)Alexandru Iosup, Dick Epema • WP3Vrije Universiteit Amsterdam (VUA)Jason Maassen, Rob van Nieuwpoort • This work is the result of the collaboration of the Partners, within the CoreGRID framework.

3. Outline • Introduction • The GrenchMark framework • Experience with GrenchMark • Extending GrenchMark • Conclusions

4. Introduction • Synthetic worklods are the basis of grid schedulers comparison [our work], and of ... • Grid systems analysis • Functionality testing in grid environments • Comparing grid settings • Grid systems design and procurement • Many others … • Synthetic grid workloads • Many different types of representative Grid applications • Need a standard tool for synthetic grid workloads generation and submission

5. Representative Grid applications (1/3) • Unitary applications • Just one scheduling unit • Examples:Sequential, MPI, Java RMI, Ibis, … • Composite applications • Composed of several unitary or composite applications • Examples:Parameter sweeps, chains of tasks, DAGs, workflows, …

6. Representative Grid applications (2/3)Unitary: synthetic • Synthetic application model with CPU cycles, memory, and I/O requirements [I]I1 data elements from stdin [O]O1 data elements to stdout for eachN steps (i) # superstep [I]I2 data elements from stdin for eachM memory locations (j) [M]get item j of size S [P]compute C computation units per memory item (fmul), and store results into temp memory location [M]put S values from temp to location j [O]O2 data elements to stdout [O]O3 data elements to stdout • Also with I/O • Also parallel, with communication

7. Representative Grid applications (3/3)Unitary: Ibis jobs • No modeling, just real parallel applications[ http://www.cs.vu.nl/ibis ] • physical simulation • parallel rendering • computational mathematics • state space search • bioinformatics • data compression • grid methods • optimization

8. The KOALA scheduler • Grid scheduler[http://www.st.ewi.tudelft.nl/koala/] • Native processor and data co-allocation supportH.H. Mohamed and D.H.J. Epema, An Evaluation of the Close-to-Files Processor and Data Co-Allocation Policy in Multiclusters, IEEE Int'l Conference Cluster Computing, Sep 2004. • By PDS group, EEMCS, PDS / Virtual Laboratory for e-Science • Application-specific support • Ibis jobs • Multi-site MPI jobs • …

9. The GrenchMark framework • What’s in a name?grid benchmark→help standardizing the testing procedures, but benchmarks are too early… • GrenchMarkA systematic approach to testing Grid schedulers • A set of metrics for comparing schedulers • A set of representative Grid applications • Both real and synthetic • Easy-to-use tools to create synthetic workloads • Flexible, portable, extensible • Can also be used for testing other Grid components, …

10. The GrenchMark framework

11. GrenchMark status • Already done in Python • Generator+ Globus, KOALA generators + RSL printer • Submitter • Results analyzer(crude) • Applications • Unitary, 3 types: sequential, MPI, Ibis (Java) • +35 different applications • Composite applications: DAG-based • Ongoing work • Automated results analyzer • Better modeling capabilities

12. GrenchMark for testing KOALA… • KOALA • Non-fixed, fixed, semi-fixed jobs • Test & evaluate • Status: pre-release, supposed stable • Workloads with different jobs requirements, inter-arrival rates, co-allocated v. single site jobs… • Jobs success rate, KOALA’s overhead and bottlenecks • Results • +5,000jobs successfully run • 2 major bugs first day, +10 bugs overall (all fixed) • KOALA is officially released(full credit to KOALA developers, 10x for testing with GrenchMark)

13. … and DAS-2’s functionality … • Already done • Evaluate for KOALA + Globus + DAS-2 • jobs success rate, turnaround time, middleware overhead, types and sources of errors • Results • 5 workloads • 500 jobs A.Iosup, J.Maassen, R.V.van Nieuwpoort, D.H.J.Epema, Synthetic Grid Workloads with Ibis, KOALA, and GrenchMark, CoreGRID IW, Nov 2005 [here, now].

14. … and for analyzing, testing, and comparing grids • Grid system analysis • Performance testing / analysis • What-if analysis • Functionality testing in grid environments • System functionality testing • Periodic testing • Comparing grid settings • Single-site vs. co-allocated jobs • Unitary vs. composite jobs A.Iosup, D.H.J.Epema, GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids, 2005 (submitted).

15. Extending GrenchMark Motto:Extending GrenchMark is easy! • Write your own Job Generators • a function with a predefined name in a Python module • auto-loaded • Write your own Unit Generators • a function with a predefined name in a Python module • auto-loaded • Interface with C/C++, Ruby, Perl, Java, … • define your own protocol • Write your own printers • a function with a predefined name in a Python module • auto-loaded

16. Conclusions and future work • GrenchMark generates diverse workloads of Grid applicationseasy-to-use, flexible, portable, extensible, … • Experienceused GrenchMark to test KOALA’s functionality and performance. used GrenchMark to test some DAS Grid functionality. +5,000 jobs generated and run … and counting. • Ongoing collaboration within CoreGRIDwith U. Dortmund group (R. Yahyapour, B. Song, C. Ernemann, …) • … advertisementHave specific Grid applications types you would like to test? Test with GrenchMark!

17. Thank you! Also: H. Mohamed and W. Lammers (KOALA), P. Anita (admin) G. Wrzesinska, N. Drost, and M. den Burger (Ibis) Questions? Remarks? Observations? All welcome! Grenchmarkhttp://grenchmark.st.ewi.tudelft.nl/[ “grenchmark”]Alexandru IOSUP TU DelftA.Iosup@ewi.tudelft.nl [ “iosup”]