Workload Modeling and its Effect on Performance Evaluation

1. Workload Modelingand its Effect on Performance Evaluation Dror Feitelson Hebrew University

2. Performance Evaluation • In system design • Selection of algorithms • Setting parameter values • In procurement decisions • Value for money • Meet usage goals • For capacity planing

3. The Good Old Days… • The skies were blue • The simulation results were conclusive • Our scheme was better than theirs Feitelson & Jette, JSSPP 1997

4. But in their papers, Their scheme was better than ours!

5. How could they be so wrong?

6. Performance evaluation depends on: • The system’s design (What we teach in algorithms and data structures) • Its implementation (What we teach in programming courses) • The workload to which it is subjected • The metric used in the evaluation • Interactions between these factors

7. Performance evaluation depends on: • The system’s design (What we teach in algorithms and data structures) • Its implementation (What we teach in programming courses) • The workload to which it is subjected • The metric used in the evaluation • Interactions between these factors

8. Outline for Today • Three examples of how workloads affect performance evaluation • Workload modeling • Getting data • Fitting, correlations, stationarity… • Heavy tails, self similarity… • Research agenda In the context of parallel job scheduling

9. Example #1 Gang Scheduling and Job Size Distribution

10. Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Ousterhout, ICDCS 1982

11. Gang What?!? Time slicing parallel jobs with coordinated context switching Ousterhout matrix Optimization: Alternative scheduling Ousterhout, ICDCS 1982

12. Packing Jobs Use a buddy system for allocating processors Feitelson & Rudolph, Computer 1990

13. Packing Jobs Use a buddy system for allocating processors

14. Packing Jobs Use a buddy system for allocating processors

15. Packing Jobs Use a buddy system for allocating processors

16. Packing Jobs Use a buddy system for allocating processors

17. The Question: • The buddy system leads to internal fragmentation • But it also improves the chances of alternative scheduling, because processors are allocated in predefined groups Which effect dominates the other?

18. The Answer (part 1): Feitelson & Rudolph, JPDC 1996

19. The Answer (part 2):

20. The Answer (part 2):

21. The Answer (part 2):

22. The Answer (part 2): • Many small jobs • Many sequential jobs • Many power of two jobs • Practically no jobs use full machine Conclusion: buddy system should work well

23. Verification Feitelson, JSSPP 1996

24. Example #2 Parallel Job Scheduling and Job Scaling

25. Variable Partitioning • Each job gets a dedicated partition for the duration of its execution • Resembles 2D bin packing • Packing large jobs first should lead to better performance • But what about correlation of size and runtime?

26. Scaling Models • Constant work • Parallelism for speedup: Amdahl’s Law • Large first  SJF • Constant time • Size and runtime are uncorrelated • Memory bound • Large first  LJF • Full-size jobs lead to blockout Worley, SIAM JSSC 1990

27. “Scan” Algorithm • Keep jobs in separate queues according to size (sizes are powers of 2) • Serve the queues Round Robin, scheduling all jobs from each queue (they pack perfectly) • Assuming constant work model, large jobs only block the machine for a short time • But the memory bound model would lead to excessive queueing of small jobs Krueger et al., IEEE TPDS 1994

28. The Data

29. The Data

30. The Data

31. The Data Data: SDSC Paragon, 1995/6

32. The Data Data: SDSC Paragon, 1995/6

33. The Data Data: SDSC Paragon, 1995/6

34. Conclusion • Parallelism used for better results, not for faster results • Constant work model is unrealistic • Memory bound model is reasonable • Scan algorithm will probably not perform well in practice

35. Example #3 Backfilling and User Runtime Estimation

36. Backfilling • Variable partitioning can suffer from external fragmentation • Backfilling optimization: move jobs forward to fill in holes in the schedule • Requires knowledge of expected job runtimes

37. Variants • EASY backfilling Make reservation for first queued job • Conservative backfilling Make reservation for all queued jobs

38. User Runtime Estimates • Lower estimates improve chance of backfilling and better response time • Too low estimates run the risk of having the job killed • So estimates should be accurate, right?

39. They Aren’t Mu’alem & Feitelson, IEEE TPDS 2001

40. Surprising Consequences • Inaccurate estimates actually lead to improved performance • Performance evaluation results may depend on the accuracy of runtime estimates • Example: EASY vs. conservative • Using different workloads • And different metrics

41. EASY vs. Conservative Using CTC SP2 workload

42. EASY vs. Conservative Using Jann workload model

43. EASY vs. Conservative Using Feitelson workload model

44. Conflicting Results Explained • Jann uses accurate runtime estimates • This leads to a tighter schedule • EASY is not affected too much • Conservative manages less backfilling of long jobs, because respects more reservations

45. Conservative is bad for the long jobsGood for short ones that are respectedConservativeEASY

46. Conflicting Results Explained • Response time sensitive to long jobs, which favor EASY • Slowdown sensitive to short jobs, which favor conservative • All this does not happen at CTC, because estimates are so loose that backfill can occur even under conservative

47. Verification Run CTC workload with accurate estimates

48. But What About My Model? Simply does not have such small long jobs

49. Workload Data Sources

50. No Data • Innovative unprecedented systems • Wireless • Hand-held • Use an educated guess • Self similarity • Heavy tails • Zipf distribution