1 / 36

An Autonomic Framework in Cloud Environment

An Autonomic Framework in Cloud Environment. Jiedan Zhu Advisor: Prof. Gagan Agrawal. Outline. Introduction Motivation Application Framework Design Overview Key Components Experiments Conclusion. Outline. Introduction Motivation Application Framework Design Overview Key Components

zizi
Download Presentation

An Autonomic Framework in Cloud Environment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. An Autonomic Framework in Cloud Environment Jiedan Zhu Advisor: Prof. GaganAgrawal

  2. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Conclusion

  3. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Conclusion

  4. Introduction • Cloud Computing • various computation and storage resources • pay-as-you-go • User Constraints • Execution Time • Cost • Problems • under utilization most of the time • longer execution • more expensive than as expected

  5. Main Challenges • Possible Solution • server consolidation ------ task consolidation • live migration ------ light-weighted migration • Our Work • an autonomic framework • Three techniques for three kinds of prior knowledge • Our Goals • Keep the application to complete within the time constraint • Keep the cost within the cost budget

  6. Contributions • Our Contributions • our system can save the cost up to 59% and more cost-efficient compared to the case when there is no resource scheduling • effective and adaptive on different workflow structures • performs better with the prior knowledge of CPU and memory requirements of tasks

  7. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Related Work • Conclusion

  8. Motivation Application • Volume Rendering • DAG-based Workflow • Parallelism • Constraints • machine capacities • resource contention • varying time constraint & cost budget

  9. Motivation Application • Figures and level

  10. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Related Work • Conclusion

  11. Framework Overview Component 1 Component 2 Component 3 Component 4

  12. Key Components 1 • Task Monitoring Agent • task status information • CPU usage, memory usage, iteration #, iteration time • checkpoints for each task • paths of input and output data • parameters for workflow • intermediate states such as iteration variable

  13. Key Components 2 • Progress Analysis Module • analyze the execution progress • workflow-specific prior knowledge A: CPU and memory requirements of tasks • initial assignment plan B: iteration structures of the workflow C: iteration structures of the tasks

  14. Progress Estimation -- A • CPU and Memory Requirements of Tasks • wocExecTime, wocTaskTime • pastTime, e.g. 500 sec pastTime current level is 2 estLevelTimet future level 3: task 4 wocTaskTime is 100sec, task5 wocTaskTime is 10 sec, task 6 is 300 sec, so estLevelTime3 is 100 x 2.5 = 250 sec, estLevelTime4 is 300 x 2.5 = 750 sec e.g. reqCPU: 50%, curCPU: 20%, so the ratiot is 2.5 estFutureTimei+1n e.g. ratiot is 2.5, wocTaskTimet is 300 sec, estTimet is 750 sec, only 1 task on current level 2, so estLevelTimeiis 750 sec estTaskTimet Total is 500 + 750 + 250 + 750 = 2250 sec

  15. Progress Estimation -- B • Iteration Structures of The Workflow • the jth iteration, total iterations is k • wocLevelTimei • pastLevelTime1i e.g. 500 sec pastLevelTime1i current level is 3 estLevelTimel estFutureTimei+1n e.g. total is 3 iterations, now it is the 1st iteration, pastLevelTime is for both level 1 and level 2.reqLevelTime1is 150 sec andreqLevelTime2is 250 sec. so ratio1i is 500 / 400 = 1.25 future level is level 4, reqLevelTime4is 250 sec, so estLevelTime4 is 312.5 sec e.g. current level is level 3 andreqLevelTime3is 300 sec, so estLevelTime3 is 375 sec The time for the 1st iteration is 500 + 375 + 312.5 = 1187.5 sec, so total is 3562.5 sec

  16. Progress Estimation -- C • Iteration Structures of Tasks • wocLevelTimeisuppose no iteration structures of workflow • remainIterNumt, avgTPerItert,pastTime e.g. 500, 0.02 sec, 500 sec estComTime1i current level 2 estLevelTimel future level 3: reqLevelTime3is 100sec, and for level 4, estLevelTime4is 300 sec, so estLevelTime3 is 100 x 1.275 = 127.5 sec, estLevelTime4 is 300 x 1.275 = 382.5 sec, so estFutureTime34 is 510 sec estFutureTimei+1n e.g. the remaining execution time for task 3 is 500 x 0.02 = 10 sec. Only 1 task on level 2, so the completion time for both level 1 and 2 is 500 + 10 = 510 sec.reqLevelTime1andreqLevelTime2 are 150 sec and 250 sec, so ratio1i is 1.275 Total is 510 + 510 = 1020 sec.

  17. Progress Estimation

  18. Key Components 3 • Scheduling Module • Greedy Algorithm • if the time constraint can not be satisfied • reschedule the instances • if the cost budget can not be satisfied while the time constraint is satisfied • consolidate the tasks and reduce the number of instances vm1 vm2 New vm

  19. Key Components • Migration Module • light-weighted checkpoints ------ migration overhead is small • timing for migration ------ 10 second point • keep data dependencies and resume the communication ------ global address book

  20. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Related Work • Conclusion

  21. Experiment Design • Experiment Goals • system effectiveness evaluation • system performance comparison under different workflow-specific prior knowledge • Experiment Environment • instance type ------ c1.medium • 2 virtual cores • 1.7GB memory • Moderate I/O performance • pricing • $0.17 / hour ------ $0.17 / 10 seconds

  22. Experiment Design • Real Application • Volume Rendering • Synthetic Workflows • synthetic workflow 1 • synthetic workflow 2 • synthetic workflow 3

  23. Experiment Design • Synthetic workflow 1 • Number of parallelism is static • No iteration structures of workflow

  24. Experiment Design • Synthetic workflow 2 • the number of parallelism is varying • no iteration structures of the workflow

  25. Experiment Design • Synthetic workflow 3 • Iteration structures for both workflow and tasks

  26. Experiment 1 • System Effectiveness Evaluation • our system vs. no scheduling • on synthetic workflow 1 and 2

  27. Experiment 1

  28. Experiment Results • Experiment Conclusion • our system can save up to 59% cost and more cost-efficient compared to the case when there is no resource scheduling • effective ------ satisfying all user requirements • adaptive to workflows of different structures

  29. Experiment 2 • system performance comparison under different workflow-specific prior knowledge vrIter performance-price ratio comparisons vrCM

  30. Experiment 2 performance-price ratio comparisons syn1CM syn1Iter syn2CM syn2Iter performance-price ratio comparisons

  31. Experiment 2 syn3CM syn3Iter performance-price ratio comparisons

  32. Experiment Results • Experiment Conclusion • With the prior knowledge with CPU and memory requirements of task, our system performs better in terms of smoothness and performance-price ratio than with other prior knowledge • may benefit from initial assignment plan

  33. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Related Work • Conclusion

  34. Related Work • Amazon web services. http://aws.amazon.com/. • Y. Ajiro and A. Tanaka. Improving packing algorithms for server consolidation. In CMG-CONFERENCE-, volume 2, page 399. Computer Measurement Group; 1997, 2007. • L. Chen, Q. Zhu, and G. Agrawal. Supporting dynamic migration in tightly coupled grid applications. In SC 2006 Conference, Proceedings of the ACM/IEEE, pages 28–28. IEEE, 2006. • Q. Zhu and G. Agrawal. Resource provisioning with budget constraints for adaptive applications in cloud environments. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing, pages 304–307. ACM, 2010. • Q. Zhu, J. Zhu, and G. Agrawal. Power-aware consolidation of scientific workflows in virtualized environments. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–12. IEEE Computer Society, 2010.

  35. Outline • Introduction • Motivation Application • Framework Design • Overview • Key Components • Experiments • Related Work • Conclusion

  36. Conclusion • Autonomic framework in the Cloud Environment • Three techniques for three kinds of prior knowledge • Task consolidation and light-weighted migration • Effective, adaptive and save the cost up to 59%

More Related