Parallel Programming on EGEE: Best practices

1. Parallel Programming on EGEE:Best practices Gergely Sipos MTA SZTAKI

2. Outline • Parallel computing architectures • Supercomputers, clusters, EGEE grid • Functional vs data parallelism • Patterns and best practices for data parallelism • From jobs to master slave to workflow

3. Traditional distributed architectures: Shared memory architecture CPU CPU Memory CPU CPU • Multiple processors operate independently but share the same memory resources • Only one processor can access the shared memory location at a time • Mutual exclusion provided by at system level • Synchronization achieved by controlling tasks' reading from and writing to the shared memory • Typical architecture of supercomputers

4. Traditional distributed architectures:Distributed memory architecture • Multiple processors operate independently, each has its own private memory • Data is shared across a network using message passing • User responsible for synchronization using message passing • Typical architecture of clusters Memory CPU Memory CPU Network CPU Memory CPU Memory

5. EGEE architecture Storage Elements • LFC catalog • AMGA database • User Interface • . . . • Some kind of distributed memory system: • Multiple processors operate independently, each has its own private memory (HDD & memory in a Computing element) • Direct communication between CEs is not possible (such as MPICH) • Some kind of shared memory system: • Central servicesto share data between CEs (between jobs)(e.g. Storage elements) • Communicating through central services must be handled at user level • No mutual exclusion, locking, etc. Computing Element Memory,HDD Memory,HDD Computing Element Network „Sharedmemory”services Memory,HDD Computing Element Computing Element Memory,HDD

6. Functional Vs Data parallelism • Functional Decomposition (Functional Parallelism) • Decomposing the problem into different jobs which can be distributed to multiple CEs for simultaneous execution • Different code run on different CEs • Good to use when there is not static structure or fixed determination of number of calculations to be performed • Domain Decomposition (Data Parallelism) • Partitioning the problem's data domain and distributing portions to multiple instances of the same job for simultaneous execution • Same code runs on different CEs processing different data • Good to use for problems where: • data is static (e.g. factoring, solving large matrix or finite difference calculations, parameter studies) • dynamic data structure tied to single entity where entity can be subsetted (large multi-body problems) • domain is fixed but computation within various regions of the domain is dynamic (fluid vortices models) • > 90% of grid applications employ data parallelism (parameter study)

7. Functional parallelism The problem Job 3on Computing Element #3 Job 4on Computing Element #4 Job 1on Computing Element #1 Job 2on Computing Element #2 Same problem size does not guarantee equal execution time Same problem size does not guarantee equal execution time time

8. Intra-job communication The problem Memory, HDD Memory, HDD Job 1on Computing Element #1 Job 2on Computing Element #2 Centralservice e.g. GFAL API or lcg-* for Storage Elements AMGA API for AMGA database LFC API for LFC catalog Sandboxes for UI time

9. Data parallelism The problem Job 2on Computing Element #2 Job 4on Computing Element #4 Job 1on Computing Element #1 Job 3on Computing Element #3 Same problem size does not guarantee equal execution time on the Grid

10. Data parallelism: Master-slave paradigm • User process running on the UI or • on a central server such as • WMS • P-GRADE Portal server • GANGA server • GridWay server Master job Local input Slave job Slave job Slave job Slave job Results Final result

11. Structure of the master Master job Generate inputs Inputs Spawn slaves Job submit Monitor slaves Slave job Slave job Slave job Slave job Check job status Collect results Get job output Results Generate final result Final result

12. Data distibution techniques • One Dimensional Data Distribution Block Distribution Cyclic Distribution To slave 1 Parameter p n > 3 jobs To slave 2 Parameter p3 jobs To slave 3 • Two Dimensional Data Distribution Cyclic blockBlockCyclic Block Block Parameter q Parameter p

13. Choose job number carefully • Less jobs  long jobs: • Smaller submission overhead • Middleware overhead 5-10 minutes / job • Waiting queue overhead 0-X minutes / job  depends on the VO • Unequal utilization of resources • Slow and fast resources must do the same amount of work • More jobs  short jobs: • Better load balancing • Faster machines do more • Overall execution time can be shorter • Submission overhead is bigger

14. Distribution of large data sets • Slaves receives only data reference from master and download real data from Storage Element, AMGA, etc. • Slaves put results into Storage Elements, AMGA, etc. and return references Inputs Master job Generate local inputs Centralservice Referencese.g. LFNs Spawn slaves Job submit Monitor slaves Slave job Slave job Slave job Slave job Check job status Collect local results Get job output Referencese.g. LFNs Generate result Centralservice Results

15. Multi-level master-slave Master job Generate inputs Input Spawn slaves Job submit Monitor slaves Slave job Slave job Slave job Slave job Check job status Collect results Get job output Results Generate inputs Master job Input Spawn slaves Job submit Monitor slaves Slave job Slave job Slave job Slave job Check job status Collect results Get job output Results Generate final result Final result

16. Complex master-slave Master job Generate inputs input Spawn slaves Monitor slaves Slave job Slave job Slave job Slave job Collect results results Generate inputs input Spawn slaves Monitor slaves Slave job Slave job Slave job Slave job Collect results results Generate inputs input Spawn slaves Monitor slaves Slave job Slave job Slave job Slave job Collect results results Generate final result Final result

17. Complex master-slave = workflow Master job Generate local inputs Workflow manager input Spawn slaves Monitor slaves 2 input Slave job Slave job 2 input Slave job Slave job Collect local results results Generate local inputs input Spawn slaves 2 input 2 input Monitor slaves Slave job Slave job Slave job Slave job Collect local results results Generate local inputs 2 input 2 input input Spawn slaves Monitor slaves Slave job Slave job Slave job Slave job Collect local results results Generate result Final result

18. Workflow managers • Mechanisms to tie pieces of application together in standard ways • Better than doing it yourself • workflow systems handle many of the gritty details • you could implement them yourself • you would do it very badly (trust me) • useful 'additional' functionality beyond basic plumbing such as • Failure management • Resubmission • Data conversion • Different requirements per scientific discipline or by application • Support for multiple levels of parallelization • Data semantics and / or Control flow semantics • Monitoring (especially for long-running workflows) • . . .

19. Microsoft WWF Moteur NetWeaver Oakgrove's reactor ObjectWeb Bonita OFBiz OMII-BPEL Open Business Engine Oracle's integration platform OSIRIS OSWorkflow OpenWFE Q-Link Pegasus Pipeline Pilot Platform Process Manager P-GRADE PowerFolder PtolemyII Savvion Seebeyond Many workflow systems for different grid middleware • Askalon • Bigbross Bossa • Bea's WLI • BioPipe • BizTalk • BPWS4J • Breeze • Carnot • Con:cern • DAGMan • DiscoveryNet • Dralasoft • Enhydra Shark • Filenet • Fujitsu's i-Flow • GridAnt • Grid Job Handler • GRMS (GridLab Resource Management System) • Sonic's orchestration server • Staffware • ScyFLOW • SDSC Matrix • SHOP2 • Swift • Taverna • Triana • Twister • Ultimus • Versata • WebMethod's process modeling • wftk • XFlow • YAWL Engine • WebAndFlo • Wildfire • Werkflow • wfmOpen • WFEE • ZBuilder …… • GWFE • GWES • IBM's holosofx tool • IT Innovation Enactment Engine • ICENI • Inforsense • Intalio • jBpm • JIGSA • JOpera • Kepler • Karajan • Lombardi • Microsoft WWF

20. Microsoft WWF Moteur NetWeaver Oakgrove's reactor ObjectWeb Bonita OFBiz OMII-BPEL Open Business Engine Oracle's integration platform OSIRIS OSWorkflow OpenWFE Q-Link Pegasus Pipeline Pilot Platform Process Manager P-GRADE PowerFolder PtolemyII Savvion Seebeyond Many workflow systems for different grid middleware EGEE Biomed community • Askalon • Bigbross Bossa • Bea's WLI • BioPipe • BizTalk • BPWS4J • Breeze • Carnot • Con:cern • DAGMan • DiscoveryNet • Dralasoft • Enhydra Shark • Filenet • Fujitsu's i-Flow • GridAnt • Grid Job Handler • GRMS (GridLab Resource Management System) • Sonic's orchestration server • Staffware • ScyFLOW • SDSC Matrix • SHOP2 • Swift • Taverna • Triana • Twister • Ultimus • Versata • WebMethod's process modeling • wftk • XFlow • YAWL Engine • WebAndFlo • Wildfire • Werkflow • wfmOpen • WFEE • ZBuilder …… • GWFE • GWES • IBM's holosofx tool • IT Innovation Enactment Engine • ICENI • Inforsense • Intalio • jBpm • JIGSA • JOpera • Kepler • Karajan • Lombardi • Microsoft WWF gLite WMS EGEE related DILIGENT project

21. During the course • gLite WMS • Parametric jobs (master-slave) • DAG (workflow) • GANGA • Parameter studies (master-slave) • P-GRADE Portal • Workflows • Parameter studies (master-slave) • Workflow based parameter studies

22. Thank you Questions?