1 / 30

CSF4 Meta-Scheduler Tutorial

CSF4 Meta-Scheduler Tutorial. 1st PRAGMA Institute Zhaohui Ding zhding@ucsd.edu or zhaohui.ding@email.jlu.edu.cn College of Computer Science & Technology Jilin University National Biomedical Computing Resource, University of California, San Diego. Agenda.

pelham
Download Presentation

CSF4 Meta-Scheduler Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CSF4 Meta-Scheduler Tutorial 1st PRAGMA Institute Zhaohui Ding zhding@ucsd.edu or zhaohui.ding@email.jlu.edu.cn College of Computer Science & Technology Jilin University National Biomedical Computing Resource, University of California, San Diego

  2. Agenda • Meta-scheduler & CSF4 Introduction • CSF4 Architecture • CSF4 Functionalities • Future Work • Demo and Practice

  3. What is Meta-Scheduler • Resource Allocation & Management • Heterogeneous • Distributed • Dynamic • Local Scheduler VS Meta-scheduler

  4. Local Scheduler VS Meta-Scheduler

  5. Meta-Scheduler VS Local Scheduler • Local Scheduler • LSF (Load Sharing Facility) • PBS (Portable Batch System) • SGE (Sun Grid Engine) • Condor • IBM Loadleveler • Meta-Scheduler • CSF • Maui (Silver) • Gridway • Nimrod-G • Condor-G

  6. CSF4 • What is CSF Meta-Scheduler • Full Name: Community Scheduler Framework • CSF4 contains a group of grid services host in GT4 • CSF4 is a full WSRF compliant meta-scheduler. • Open Source project and can be accessed at http://sourceforge.net/projects/gcsf • Developed by Lab. of Distributed Computing and System Architecture, Jilin University, China • CSF4 has been added to Globus Toolkit 4 as an Execution Component

  7. CSF4 in Globus Toolkit 4

  8. A typical deployment

  9. What CSF4 Can Do? • Basic Functionalities • Submit jobs to Grid without Specifying Cluster • Monitor and Control Jobs • Provide Queuing Service • Schedule jobs and resource by custom-built polices • CSF4 Portlet (A Web browser based User Interface)

  10. What CSF4 Can Do? (cont.) • Advanced Functionalities • Multiple Domains Resource Information Sharing • Multiple scale resource scheduling policies • Automatic user credentials delegation • Automatic data-staging • Extensible scheduling framework • Supporting grid parallel jobs (MPI&MPICH-G2)

  11. CSF4 – Architecture

  12. CSF4 – Architecture User view

  13. Local Scheduler And Infrastructure Supported by CSF4 • Local Scheduler Supported • LSF • PBS • SGE • Condor • Infrastructure Supported • Globus Toolkit 4 • Globus Toolkit 2

  14. CSF4 – Functionalities Scheduling Plug-in Framework • Designed For Queuing Service • Provide A set of policies • Customizable • Extensible

  15. Existent Scheduling Policies • FCFS (First Come First Serve) round-robin • Default policy • Throttle • Restrict the number of jobs in a scheduling cycle • Array Job Plug-in • Design for life science applications (such as AutoDock, BLAST) • MPICH-G2 Plug-in (under-developing) • The plug-in guarantee the synchronized resource allocation can be successful • Data intensive applications plug-in (under-developing)

  16. Schedule plug-in & scheduling policies • Each policy is implemented inside a scheduling plugin module • A queue can load multiple plugin modules

  17. Resource Information Sharing • A MDS information provider for CSF4 • Multiple CSF4 can share the resource information

  18. CSF4 – Functionalities (cont.) • Deploy Multiple CSF4 in a Grid Community

  19. Array Job • AutoDock and Blast-like applications • A large number of sub-jobs. • Execute same binary • Different input/output files

  20. Array Job (cont.) • Advantages • Submit job only once • Save submission time and memory storage

  21. Data Staging • Manual Data Staging • Which clusters I can use? • Which clusters my jobs will running on? • Where is the output data? • When will the job finish, so that I can stage-out the output data?

  22. Manual Data Staging • Without Meta-Scheduler

  23. Automatic Data Staging • With CSF4 Automatic Data Staging

  24. Integrate CSF4 with Gfarm • With CSF4 Automatic Data-Staging and Gfarm

  25. Application Based Scheduling

  26. CSF4 User Interface • CSF4 Portal

  27. CSF4 User Interface • CSF4 Command Line

  28. Under-Developing work

  29. Demo & Practice • https://www.nbcr.net/pub/wiki/index.php?title=CSF4_Tutorial_PRAGMA13

  30. Thank you감사합니다ありがとうございます謝謝谢谢

More Related