KARMA with ProActive Parallel Suite 12/01/2009 Air France, Sophia Antipolis

1. KARMA with ProActive Parallel Suite12/01/2009Air France, Sophia Antipolis Solutions and Services for Acceleratingyour Applications

2. ProActive Parallel Suite Speed-up applications by distributing them over a set of resources The result of 9 years of R&D at INRIA

3. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

4. ProActive Technology Integration • Simple and Seamlessintegration • Installation : one directory • Deployment : one XML file (GCM Deployment) • Possible automated agent-based deployment • Support for common standards • Connection/Acquisition: Ssh, Rsh, LSF, PBS, SGE,… • Communication: RMI, RMI/SSH, HTTP, Ibis,… • Export any activity as a WebService • J2EE (in and out) • Validated on common JDKs 1.5, 1.6 and 1.7 • Activities are monitored through standard JMX • Compliant with common monitoring tools (TIBCO Hawk)

5. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

6. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

7. ProActive Scheduling Managed environment for distributed computation

8. Scheduler / Resource Manager Overview • Multi-platform Graphical Client (RCP) • File-based or LDAP authentication • Static Workflow Job Scheduling, Native and Java tasks, Retry on Error, Priority Policy, Configuration Scripts,… • Dynamic and Static node sources, Resource Selection by script, Monitoring and Control GUI,… • ProActive Deployment capabilities : Desktops, Clusters, ProActive P2P,…

9. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

10. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

11. Parallelism with ProActive Scheduling • Computations are defined as Jobs made of Tasks • Bag of Tasks • Tasks Flow • ProActive application Task 2(input 2) Task 1(input 1) Task 1(input 1) res2 res1 Task 3(res1,res2) Functional // Technical // Task 1(input 1) Task 2(input 2) Task N(input N) …

12. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

13. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

14. Defining Tasks Tasks can be defined in Java or as Native Application • Dynamic Task creation through Java API JavaTaskaTask = new JavaTask(); aTask.setName("task 1"); aTask.setRerunnable(2); aTask.setRestartOnError(RestartMode.ELSEWHERE); aTask.addArgument("foo",new Boolean(true)); aTask.setExecClassName("com.activeeon.Compute"); aTask.setPreScript("/path/to/script_file"); • Can attach pre/post script to a task • Any JSR-223 supported language (JS, Python, Ruby,…) • Used for configuration and data transfer  Data are handled separately

15. Defining Jobs Jobs can be defined in Java or XML • Dynamic job creation through Java API TaskFlowJob job = new TaskFlowJob(); job.setName("job_name"); job.setPriority(JobPriority.NORMAL); job.setCancelOnError(true); job.addTask(new NativeTask(cmd));  Can easily develop tools that generate and submit jobs • Specialized • Automated • Static description in XML descriptor  Can be defined by any non specialized user

16. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

17. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

18. Executing Jobs • Submission through… • Java API • Graphical interface • Command line • Recovery is fully configurable with User/System level distinction • Host failure : automatically re-dispatched with needed input • Number of retries bounded by the administrator • Taskfailure : can be either… • Automatically re-dispatched with needed input • Number of retries bounded by the user • Aborted (Task) : exception is transmitted as task result • Aborted (Job) : job is fully canceled • Scheduler itself is recoverable from underlying database

19. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

20. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

21. Monitoring Jobs • System and ProActive level monitoring • Through standard JMX • Provide generic tool for monitoring and reporting • Compliant with common monitoring tool

22. Monitoring Jobs • System and ProActive level monitoring • Through standard JMX • Provide generic tool for monitoring and reporting • Compliant with common monitoring tool • Scheduler level monitoring • On-going porting from MVC to JMX

23. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

24. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

25. Controlling Jobs • Fine grain control through GUI and API Job level (User) Scheduler level (Admin)

26. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

27. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

28. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab

29. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab • User-level hooks to (very) easily specialize the Scheduler • Plug-in for RCP client • e.g. ActiveEon’s Gold module • Result preview mechanism • Can preview result in the scheduler GUI (textual/graphical) • Can attach any external application to a task for result display

30. Specializing the Scheduler • Business Level specialization and automation • ActiveEon provides business specific interface for the Scheduler • Bio-technology (Gold, MapRead) • Finance • MatLab/Scilab • User-level hooks to (very) easily specialize the Scheduler • Plugin for RCP client • e.g. ActiveEon’s Gold module • Result preview mechanism • Can preview result in the scheduler GUI (textual/graphical) • Can attach any external application to a task for result display

31. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

32. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

33. ProActive Resource Manager • Rely on ProActive Programming features • GCM deployment standard • Agent based deployment • Support for many LAN / WAN / Grid / Cloud standards… Cluster MultiCPU/Core Clusters / Grille Clouds / RAS Desktop Grid Straightforward Upgrade

34. ProActive Resource Manager • Rely on ProActive Programming features • GCM deployment standard • Agent based deployment • Support for many LAN / WAN / Grid / Cloud standards… • Deal with resources heterogeneity with Selection Scripts • User defined : no need for system reconfiguration • Any JSR-223 supported language (JS, Python, Ruby,…) • Can rely on dynamic decision or on static host description • Optimized (probabilistic selection of candidates)

35. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

36. KARMA performances constraints • KARMA processes computation needs refer to : • HEAVY PROCESS (Batch long time) • LIGHT PROCESS (Interactive short time) • Processes are expressed according to the following features: • Triggering mode (start and kill) • Parallelization capability (functional / technical) • Re-run ability (fault tolerance / recovery) • Performance (elapse time) • Type (interactive, schedule, event, exceptional, administrative) • Data volume (light, medium, heavy) • Manage resources consumption • Availability • Resources optimization (CPU, Core, Memory) • Strategy / Policy (Arbitration resources and tasksrequestprioritization) • Fault error management • Open system • Interoperability (upper and lower stream) • API (language, completeness, reliability) • Tools (friendliness, eclipse integration ability,..) • Parallelization • Multiple processes capability (functional) • Parallelization programming capability (technical).

37. Thank you for your attention • Demo