Surviving in the Wild : Teaching and Training for the Parallel Future

1. Surviving in the Wild:Teaching and Training for the Parallel Future Dick Brown St. Olaf College SPLASH Educators’ and Trainers Symposium October 24, 2011

2. Overview • Review of the need for parallelism • Strategies we need • What to teach • How to teach it • How to get it taught • Surviving in the wild of Parallelism With interludes, to be announced…

3. The need for parallelism • Note: I will use parallel as a generic for concurrent, parallel, distributed, cloud, accelerator (e.g., GPGPU), etc. • Question: Why do we need parallelism at the programming level? • Hint: It’s not “because it’s there”(… although desktop applications do tend to find ways to use ever greater power per dollar)

4. The need for parallelism • Answer: Scale

5. The need for parallelism • Answer: Scale • Cloud applications + = 30,000,000,000,000,000

6. The need for parallelism • Answer: Scale • Cloud applications • Scientific applications: Particle-level simulation of turbulance is exascale • Can’t achieve exascale performance withoutmany cores (Berkeley “walls”), accelerators

7. Challenges for industry • Technology: • Heterogeneous computing (CPU + accelerators) • Sophisticated “on the fly” runtime systems • “Wall” of memory hierarchy vs. on-chip access

8. Challenges for industry • Technology: • Heterogeneous computing (CPU + accelerators) • Sophisticated “on the fly” runtime systems • “Wall” of memory hierarchy vs. on-chip access • Examples • AMD Fusion System Architecture: CPU+GPU • Intel MIC (Many Integrated Cores): 50+ CPUs on a chip, as a cluster-like accelerator

9. Challenges for industry • Technology: • Heterogeneous computing (CPU + accelerators) • Sophisticated “on the fly” runtime systems • “Wall” of memory hierarchy vs. on-chip access • Programming models: • Higher level; more “human-centric” • Scalable • Versatile

10. Challenges for Education/Training We want to prepare our students for whatthey’ll need, before the demand explodes, but • What are the enduring principles? • Technologies, (hence) tools change rapidly! • (Educators:) Change the curriculum???

11. A wild ecosystem • Industry/Academia

12. A wild ecosystem • Industry/Academia • Learning curve/Rapid change

13. A wild ecosystem • Industry/Academia • Learning curve/Rapid change • Principles/Practices

14. A wild ecosystem • Industry/Academia • Learning curve/Rapid change • Principles/Practices • Teaching/Research • New research discoveries in technology and programming models need to get into the curriculum yesterday

15. A wild ecosystem • Industry/Academia • Learning curve/Rapid change • Principles/Practices • Teaching/Research And, it’s coming fast! • Took OOPSLA 20 years to become SPLASH…We can’t wait that long We need strategy!

16. Strategies to be found • What to teach • How to teach it • How to get it taught ITiCSE 2010working group, Strategies for Preparing Computer Science Students for the Multicore World

17. What to teach • Parallel computing has a head start: ACM/IEEE Curriculum ’91 • 3 required hours on parallel algorithms • 3 required hours on distributed and parallel programming language constructs, with hands-on practice • Ada,   Concurrent  Pascal,  Occam,  or  Parlog (Was not universally embraced…)

18. What to teach • Parallel computing has a head start: ACM/IEEE Curriculum ’91 • 3 required hours on parallel algorithms • 3 required hours on distributed and parallel programming language constructs, with hands-on practice But, ten years later… • ACM/IEEE Curriculum ’01 • 0 required hours of parallel algorithms • No mention of programming language constructs • Replaced by: “net-centric computing,” etc.

19. NSF/TCPP Curriculum Standards Initiative in Parallel and Distributed Computing – Core Topics for Undergraduates Sushil K. Prasad, IEEE TCPP Chair, Georgia State University Richard LeBlanc, Seattle University, ACM Education Council Charles Weems, University of Massachusetts, Amherst Alan Sussman, University of Maryland Arnold Rosenberg, Northeastern and Colorado State University Andrew Lumsdaine, Indiana University Curriculum Initiative Website: linked through tcpp.computer.org http://www.cs.gsu.edu/~tcpp/curriculum/index.php

20. Who are we? • Chtchelkanova, Almadena - NSF • Dehne, Frank - University of Carleton, Canada • Gouda, Mohamed - University of Texas, Austin, NSF • Gupta, Anshul - lBM T.J. Watson Research Center • JaJa, Joseph - University of Maryland • Kant, Krishna - NSF, Intel • La Salle, Anita - NSF • LeBlanc, Richard, University of Seattle • Lumsdaine, Andrew - Indiana University • Padua, David- University of Illinois at Urbana-Champaign • Parashar, Manish- Rutgers, NSF • Prasad, Sushil- Georgia State University • Prasanna, Viktor- University of Southern California • Robert, Yves- INRIA, France • Rosenberg, Arnold- Colorado State University • Sahni, Sartaj- University of Florida • Shirazi, Behrooz- Washington State University • Sussman, Alan - University of Maryland • Weems, Chip, University of Massachussets • Wu, Jie - Temple University

21. Specifying Curriculum Recommendations – NSF/TCPP Approach • Identify topics in four existing areas: architecture, algorithms, programming, and cross-cutting topics • For each topic, recommend • Bloom level • “Hours” of coverage • Suggested learning outcome • Possible core course for coverage • Focus: First two years

22. Bloom Levels Use first three levels for recommended core topics • K= Know the term/recall definition (basic literacy) • C = Comprehend so as to paraphrase/illustrate • A = Apply it in some way (requires operational command) • N = Not in core (but may be useful in elective or advanced courses)

23. K: know term C: paraphrase/illustrate A: apply Example • Parallel and Distributed Models and Complexity • Costs of computation

24. Programming • Assume some conventional (sequential) programming experience • Key is to introduce parallel programming early to students • Four overall areas • Paradigms – By target machine model and by control statements • Notations – language/library constructs • Correctness – concurrency control • Performance – for different machine classes

25. Parallel Programming Paradigms (Selections) • By target machine model • Shared memory (Bloom classification A) • Distributed memory (C) • Client/server (C) • Hybrid (K) – e.g., CUDA for CPU/GPU • By control statements • Task/thread spawning (A) • Parallel Loop (C)

26. How to Read the Proposal • Oh no! Not another class to squeeze into our curriculum!

27. How to Read the Proposal • Oh yes! Not another class to squeeze into your curriculum!

28. How to Read the Proposal • Oh yes! Not another class to squeeze into your curriculum! • Draft curriculum released Dec 2010 (tcpp.computer.org)

29. Enduring skills? • Since the tool set is subject to change at any time, how much investment in those skills? • Many parallel languages and features have come and gone • Need hands-on experience for effective learning. • Anything may suddenly emerge as important • Python as a prototyping language for HPC

30. A candidate for addition Patterns of parallel programming

31. Patterns, a candidate for addition • Background • “Gang of Four” book

32. Patterns, a candidate for addition • Background • “Gang of Four” book, 1994 • Doug Lea, Concurrent programming in Java: Design principles and patterns, 1999 • Tim Mattson, et al, Patterns for Parallel Programming, 2005 • Kurt Kreutzer and Berkeley Parlab, the Dwarves Motifs • Kreutzer and Mattson, OPL (parlab.eecs.berkeley.edu/wiki/patterns)

33. Patterns, a candidate for addition Why patterns? • They capture reusable units of expert problem-solving strategy • Thus, they provide novices with a way to acquire expertise • Many are supported by tools • Loop parallel, Message passing, Map-reduce, …

34. How to teach it • Agree with NSF/TCPP Initiative, that parallelism should be taught early and often • Scratch team kept concurrent scripts, because users “not surprised that a sprite can do several things at once” • Lessons of Vishkin’s “Peanut Butter Sandwich” exercise

35. CSinParallel project • Add parallelism early and often at all levels • Incremental, flexible approach via modules • Sharing within our community

36. CSinParallel project • Modular Approach • Short units (1-3 days) • Identified learning objectives • Self-contained • Flexible for use in various courses and curricula • Make software/libraries more accessible • Parallel Platform Packages, Resources • Share, discuss and help as a community • http://csinparallel.org

37. CSinParallel project Some selected module topics • Introductory: • Map-Reduce computing for CS1 using WebMapReduce • Concurrent access to data structures in Java or C++ • Multicore programming with Intel’s Manycore Testing Lab • Intermediate: • Introduction to parallel computing concepts • Concurrency strategies in programming languages • Parallel sorting algorithms

38. Module: WebMapReduce

39. Module: MTL with OpenMP • Intel’s Manycore Testing Lab • Module • #pragma omp parallel for num_threads(threadct) \shared (a, n, h, integral) private(i) • reduction(+: integral)  

40. CSinParallel • We seek collaborators and contributors

41. Patterns Methodology • Kreutzer and Mattson OPL not only provides a catalog of patterns, but also a software problem-solving methodology

42. Patterns Methodology • Kreutzer and Mattson OPL not only provides a catalog of patterns, but also a software problem-solving methodology • Purposes: • Education • Communication • Design

43. How to get it taught • Pressures on the professor • “Oh no! Not another course to squeeze…” • So, take an incremental spiral approach(agreeing with NSF/TCPP) • Small changes in curriculum in many places • Revisit challenging issues • Students come to think of parallelism as natural part of computation • Spiral approach is pedagogically effective

44. How to get it taught Incentives • Microgrants: small (e.g., $1500) amounts for contributing first steps in teaching parallelism • Intel Academic Community (intel.com/AcademicCommunity) • Educational Alliance for a Parallel Future (eapf.org)

45. NSF/TCPP InitiativeEarly Adopter Program

46. How to obtain Early Adopter Status? • 16 Early adopters chosen for Spring term 2011 • 17 Early adopters chosen for Fall term 2011 • Next round of competition:Fall 2012; DeadlineNovember 5, 2011 • NSF/Intel funded Stipend/Honorarium • Which course(s) , topics, evaluation plan?

47. How to obtain Early Adopter Status? • Instructors for • core CS/CS courses such as CS1/2, Systems, Data Structures and Algorithms – department-wide adoption preferred • elective courses such as Algorithms, Architecture, Programming Languages, Software Engg, etc. • introductory/advanced PDC course • dept chairs, dept curriculum committee members responsible

48. How to get it taught Other supports needed • Platform availability • Support community • Educational elements • Learning objectives, assessment tools, etc.

49. Surviving in the wild ecosystem • Industry/Academia • Learning curve/Rapid change • Principles/Practices • Teaching/Research

50. Surviving in the wild ecosystem • Industry/Academia • Learning curve/Rapid change • Principles/Practices • Teaching/Research • Mine from the heritage of the past • Incremental approach • Spiral exposition • Pattern-based methods