Building Scalable Scientific Applications using Makeflow

1. Building Scalable Scientific Applications using Makeflow Dinesh Rajan and Peter Sempolinski University of Notre Dame

2. Cooperative Computing Lab University of Notre Dame http://www.nd.edu/~ccl

3. The Cooperative Computing Lab The Cooperative Computing Lab The Cooperative Computing Lab The Cooperative Computing Lab • We collaborate with people who have large scale computing problems in science, engineering, and other fields. • We operate computer systems on the O(10,000) cores: clusters, clouds, grids. • We conduct computer science research in the context of real people and problems. • We develop open source software for large scale distributed computing. • We collaborate with people who have large scale computing problems in science, engineering, and other fields. • We operate computer systems on the O(10,000) cores: clusters, clouds, grids. • We develop open source softwarefor large scale distributed computing. http://www.nd.edu/~ccl 3

4. Plan for Today’s Tutorial Our CCTools Software Makeflow, Work Queue, Parrot, Chirp Makeflow Lecture: Overview, features Tutorial:Write simple Makeflows Work Queue Lecture: Overview, features Tutorial:Write simple WQ programs

5. Science Depends on Computing!

6. The Good News:Computing is Plentiful! 6

9. Superclusters by the Hour http://arstechnica.com/business/news/2011/09/30000-core-cluster-built-on-amazon-ec2-cloud.ars

10. I have a standard, debugged, trusted application that runs on my laptop. A toy problem completes in one hour. A real problem will take a month (I think.) Can I get a single result faster? Can I get more results in the same time? Last year, I heard about this grid thing. This year, I heard about this cloud thing. 10

11. I have allocationson clusters (unlimited) + grids (limited) + clouds ($)!How do I run my applicationon those machines?

12. Should I port my program to MPI or Hadoop? Learn MPI / Hadoop Learn C / Java Re-architect Re-write Re-test Re-debug Re-certify

13. And my application looks like this…

14. Makeflow & Work Queue Easy to scale from one desktop to national scale infrastructure. Harness all available resources: desktops, clusters, clouds, grids. Portable across operating systems, storage systems, batch systems. No special privileges required.

15. Makeflow part1 part2 part3: input.data split.py ./split.py input.data out1: part1 mysim.exe ./mysim.exe part1 >out1 out2: part2 mysim.exe ./mysim.exe part2 >out2 out3: part3 mysim.exe ./mysim.exe part3 >out3 result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result

16. Work Queue Library #include “work_queue.h” while( not done ) { while (more work ready) { task = work_queue_task_create(); // add some details to the task work_queue_submit(queue, task); } task = work_queue_wait(queue); // process the completed task } http://www.nd.edu/~ccl/software/workqueue 16

17. Parrot and Chirp Ordinary Appl Filesystem Interface: open/read/write/close Parrot Virtual File System Local HTTP CVMFS Chirp iRODS Web Servers CVMFS Network Chirp Server iRODS Server 17

18. http://www.nd.edu/~ccl/software/manuals

19. Source code in GitHub http://github.com/cooperative-computing-lab/cctools

20. Makeflow & Work Queue Federate/harness all available resources: desktops, clusters, clouds, grids. Simple interfaces & API Part of CCTools software No special privileges required to install.

21. Makeflow Lecture: Outline • What is Makeflow? • Portable: One Makeflow program for SGE, Condor, PBS • How to write an application using Makeflow? • Simple rule-based syntax • How to run Makeflow? • Features, commands, using Work Queue

22. An Old Idea: Makefiles part1 part2 part3: input.data split.py ./split.py input.data out1: part1 mysim.exe ./mysim.exe part1 >out1 out2: part2 mysim.exe ./mysim.exe part2 >out2 out3: part3 mysim.exe ./mysim.exe part3 >out3 result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result 22

23. Makeflow Language - Rules Each rule specifies: a set of target files to create; a set of source files needed to create them; a commandthat generates the target files from the source files. part1 part2 part3: input.data split.py ./split.py input.data out1: part1 mysim.exe ./mysim.exe part1 >out1 out2: part2 mysim.exe ./mysim.exe part2 >out2 out3: part3 mysim.exe ./mysim.exe part3 >out3 result: out1 out2 out3 join.py ./join.py out1 out2 out3 > result out1 :part1 mysim.exe mysim.exe part1 > out1

24. You must stateall the filesneeded by the command.

25. sims.mf out.10: in.dat calib.dat sim.exe sim.exe –p 10in.data > out.10 out.20: in.dat calib.dat sim.exe sim.exe –p 20in.data > out.20 out.30: in.dat calib.dat sim.exe sim.exe –p 30in.data > out.30

26. Makeflow = Make + Workflow Provides portability across batch systems. Enable parallelism (but not too much!) Fault tolerance at multiple scales. Data and resource management. http://www.nd.edu/~ccl/software/makeflow Makeflow Local Condor Work Queue SGE 26

27. Makeflow + Batch System XSEDE Cluster Private Cluster Makefile makeflow –T sge Work Queue Makeflow Work Queue Campus Condor Pool Public Cloud Provider makeflow –T condor Local Files and Programs

28. How to run a Makeflow Run a workflow local %makeflow -T localsims.mf Run the workflow on SGE: %makeflow -T sgesims.mf Run the workflow on Condor: % makeflow -T condorsims.mf Clean up the workflow outputs: %makeflow -c sims.mf

29. Makeflow can verify if your Makeflow file is syntactically correct % makeflow -k sims.mfMakeflow: Syntax OK. Makeflow will point out syntax errors if any % makeflow -k sims.mfmakeflow: out10 is defined multiple times at out.10:1 and out.10:4 Makeflow Syntax Checker

30. Makeflow can output a makeflow file as a Dot graph. % makeflow -D dot sims.mf digraph { node [shape=ellipse,color = green,style = unfilled,fixedsize = false]; N2 [label="sim.exe"]; N1 [label="sim.exe"]; N0 [label="sim.exe"]; node [shape=box,color=blue,style=unfilled,fixedsize=false]; F3 [label = "out.30"]; F0 [label = "sim.exe"]; F5 [label = "out.10"]; F2 [label = "in.dat"]; F1 [label = "calib.dat"]; F4 [label = "out.20"]; .. .. Makeflow Visualization

31. Example App: Biocompute Portal BLAST SSAHASHRIMP EST MAKER … Progress Bar Condor Pool Generate Makefile Transaction Log Update Status Run Workflow Make flow Submit Tasks

32. Makeflow + Work Queue

33. Makeflow + Batch System XSEDE Cluster Private Cluster Makefile makeflow –T sge ??? Makeflow ??? Campus Condor Pool Public Cloud Provider makeflow –T condor Local Files and Programs

34. Makeflow + Work Queue sge_submit_workers Private Cluster XSEDE Cluster Makefile W W Thousands of Workers in a Personal Cloud W submit tasks W W W Makeflow Campus Condor Pool Public Cloud Provider W W W W W W W Local Files and Programs condor_submit_workers ssh

35. Advantages of Work Queue Scalability: Harness multiple infrastructure simultaneously. Elasticity: Scale resources up & down as needed. Data Management: Remote data caching. Data Locality: Matches tasks to nodes with data.

36. Fault Tolerance MF +WQ is fault tolerant : If Makeflow crashes (or killed), it recovers by reading log and continues where it left off. If a worker crashes, the master will detect and restart the task elsewhere. Workers can be added and removed any time during execution.

37. Makeflow and Work Queue To start the Makeflow % makeflow -T wqsims.mf Could not create work_queueon port 9123. % makeflow -T wq -p 0 sims.mf Listening for workers on port 8374… To start one worker: % work_queue_worker ccl.cse.nd.edu 8374

38. Start Workers Everywhere! Submit workers to SGE: %sge_submit_workers ccl.cse.nd.edu 837425 Submit workers to Condor: %condor_submit_workers ccl.cse.nd.edu 8374 25 Submit workers to Torque: %torque_submit_workers ccl.cse.nd.edu 8374 25

39. Keeping track of port numbersgets old fast…

40. Project Names work_queue_worker -a –N myproject makeflow … –a –N myproject Worker connect to ccl.cse.nd.edu:4057 Makeflow (port 4057) Catalog advertise query “myproject” is at ccl.cse.nd.edu:4057

41. Makeflow with Project Names Start Makeflow with a project name: %makeflow -T wq -p 0 -a -N xsede-tutorialsims.mf Listening for workers on port XYZ… Start one worker: %work_queue_worker -N xsede-tutorial Start many workers: %sge_submit_workers -N ccgrid-tutorial 5

42. http://www.nd.edu/~ccl/software/makeflow/

43. The Cooperative Computing Lab Makeflow The Cooperative Computing Lab The Cooperative Computing Lab • Portable: One program for clusters, grids, clouds • Simple syntax: inputs, outputs, command • All files needed by command must be specified • Makeflow with Work Queue • Federation, Elasticity, Data management • Project Names • Easy to remember locations of Makeflow masters • We collaborate with people who have large scale computing problems in science, engineering, and other fields. • We operate computer systems on the O(10,000) cores: clusters, clouds, grids. • We conduct computer science research in the context of real people and problems. • We develop open source software for large scale distributed computing. 43

44. Chris Hempel (TACC) David Gignac (TACC) Acknowledgements

45. Go to: http://www.nd.edu/~cclClick on “Tutorial at XSEDE 2013”

46. Click on “Tutorial” under Makeflow