1 / 13

Workflow Adaptation as an Autonomic Computing Problem

Workflow Adaptation as an Autonomic Computing Problem. Kevin Lee, Rizos Sakellariou, Norman W. Paton and Alvaro A. A. Fernandes School of Computer Science, University of Manchester {klee, rizos, norm, alvaro}@cs.man.ac.uk. Kevin Lee 25 th June 2007. Talk Overview.

heather-cox
Download Presentation

Workflow Adaptation as an Autonomic Computing Problem

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Workflow Adaptation as an Autonomic Computing Problem Kevin Lee, Rizos Sakellariou, Norman W. Paton and Alvaro A. A. Fernandes School of Computer Science, University of Manchester {klee, rizos, norm, alvaro}@cs.man.ac.uk Kevin Lee 25th June 2007

  2. Talk Overview Reasons for Adaptation in Workflows Autonomic Computing Our View of Workflows Potential Workflow Adaptations Categorising Workflow Adaptations Future work

  3. 1. Reasons for Adaptation in Workflows Execution Characteristics of Scientific Workflows • Very long running • Small delays can have large effects due to dependencies • often involve highly distributed resources • Limited control over resources • Uncertain execution times • Uncertain queue waiting times

  4. 1. Reasons for Adaptation in Workflows • scheduling of a workflow is decided before it starts executing • Using current information about the execution environment • What happens if the environment changes? • Resources disappear • Loads change • New resources appear • What if a execution resource becomes unavailable • Only real option is to Re-Submit to different resources • Reduced performance • New resources are not taken advantage of • The Execution characteristics of workflows combined with these lead to lower than potential execution performance Adaptation would be desirable. But what would workflow adaptation look like?

  5. 2. Autonomic Computing • Many systems nowadays face these issues • Building adaptive systems is hard • always done in ad-hoc ways • leads to brittle and non-reusable adaptation • To this end adaptive systems are often seen in the Autonomic Systems Community as functionally decomposable into the components: Monitor: Events from a source: log files in-memory process sensors Analyze: When an event occurs, what to do about it... Plan: After the event is detected and analysed, the system needs to determine what to do about it. Execute: Perform the necessary changes But, how do we think about workflows in these terms?

  6. 3. Our View of Workflows • To look at workflows in a generic way, we’ve adopted a view of the use of workflows as follows: • An abstract workflow: • describes the workflow at the level of tasks that perform transformations on data. • A concrete workflow: • describes the workflow at the level of actual services • file-based inputs and outputs. • Mapping an abstract workflow • choosing appropriate services for tasks • finding data sources and output files • Scheduling a concrete workflow • assigning each of the services to execution nodes. We have a more formal notation in the paper

  7. 4. Potential Workflow Adaptations • In general, adaptations can usefully be thought of as a revision of decisions made previously • Thus, based on the previous slide, workflow adaptations can be classified as either mapping or scheduling adaptations. • Adaptations can be performed for different reasons: • prospective (to improve future performance)‏ • reactive (to react to previous results)‏ • altruistic (to aid other areas of the system). • Adaptations can also affect the workflow at different • levels of granularity: • single node • some nodes • all of the workflow.

  8. 4. Potential Workflow Adaptations: Mapping Adaptations • Mapping adaptations are adaptations where the mapping from the abstract workflow to the concrete workflow changes depending on the environment. • Examples: • Change abstract node to concrete node mapping: • Reduce the number of concrete nodes for an abstract task • Increase the number of concrete nodes for an abstract task (task-splitting). • Remove an abstract node • Change data source/sink for a service See paper for further detail

  9. 4. Potential Workflow Adaptations: Scheduling Adaptations • Adaptive scheduling involves the alteration of the scheduling policy in response to changes in the environment. • Examples: • Increase the level of parallelism of a service. • Decrease the level of parallelism of a service. • Restart service. • Pause service. • Move service between execution nodes. See paper for further detail

  10. 5. Categorising Workflow Adaptations We used the MAPE functional decomposition to look at workflow adaptation The Monitoring, Analysis, Planning and Execution functional phases can be used to investigate the adaptive opportunities They provide a consistent, abstract viewpoint with which to express adaptation strategies The Various options in M,A,P,E can be arranged as follows: Monitoring Analysis Planning Execution Progress of a service Completion of a service Data consumption rate of a service Data production rate of a service Available execution nodes Load on an execution node Load on a network link Memory usage on an execution node Available services Available data resources Load Imbalance Bottleneck Potential Workflow QoS miss Execution node failure Free capacity New service available New data available Underutilised execution node Increase service parallelism Reschedule a service Replace a service Use free execution nodes Move services Change data sources Execute changes This level of understanding, combined with a adaptivity infrastructure provides a solid basis for providing adaptivity functionality

  11. 6. Current activities and Future work • Creating an infrastructure to support the Systematic Development of Adaptive Systems based on the ideas presented today • Ease the development of adaptive systems. • Support the development of better adaptive systems • Investigate the use of the infrastructure in a number of different domains • Use the infrastructure to improve the general understanding of adaptive systems • Applying the infrastructure to related domains • Simulated DAG Scheduling • Workflow processing with the Pegasus team • Concurrent business workflows • Distributed Query Processing • case studies...

  12. Project Organisation EPSRC e-Science project entitled: “An Infrastructure for Adaptive Systems Development” Kevin Lee, Norman W. Paton, Alvaro A. A. Fernandes, Rizos Sakellariou School of Computer Science, University of Manchester Oxford Road, Manchester, M13 9PL, U.K. {klee, rizos, norm, alvaro}@cs.man.ac.uk Jim Smith, Paul Watson School of Computing Science, Newcastle University Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K. {Jim.Smith, Paul.Watson}@ncl.ac.uk Jim Smith, Paul Watson School of Computing Science, Newcastle University Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K. {Jim.Smith, Paul.Watson}@ncl.ac.uk Jim Smith, Paul Watson School of Computing Science, Newcastle University Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K. {Jim.Smith, Paul.Watson}@ncl.ac.uk Jim Smith, Paul Watson School of Computing Science, Newcastle University Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K. {Jim.Smith, Paul.Watson}@ncl.ac.uk Jim Smith, Paul Watson School of Computing Science, Newcastle University Claremont Road, Newcastle upon Tyne, NE1 7RU, U.K. {Jim.Smith, Paul.Watson}@ncl.ac.uk

  13. Questions?/Comments?

More Related