1 / 14

Automate Distributed Task Management Using Python and AMQP with JobRunner

JobRunner is a robust tool designed for automating distributed tasks in cloud environments utilizing Python and AMQP. As Cloudbase experiences rapid growth with OpenStack, effective task scheduling and management is vital. JobRunner allows users to create custom tasks, providing a lightweight and extensible framework. It features a REST API, a controller, workers, and notifications, ensuring asynchronous interactions between components. Tasks include essential methods and attributes like create, delete, and state management, facilitating efficient resource utilization. Discover more at our GitHub repository.

tejana
Download Presentation

Automate Distributed Task Management Using Python and AMQP with JobRunner

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JobRunner Running Distributed tasks using Python and AMQP Gabriel Samfira, Cloud Engineer

  2. A bit of background Cloudbase has seen significant growth since starting work on OpenStack Testing is essential The need for automating all this madness has become painfully obvious

  3. JobRunner Distribute workload Allow users to create own tasks Light weight Decently extensible

  4. Components JobRunner REST API (sort of) JobRunner Controller JobRunner Worker JobRunner Notifications

  5. Design Evented worker or single threaded All interactions between components are asynchronous Tasks are created for resources Dispatcher determines the module that should be loaded Actions define the method from the module (well..actually class) that is to be run

  6. Resources At a minimum, all resources must define the following methods: create delete is_idle lock Save And the following attributes: uuid (this would be the task ID) state

  7. Tasks A few of the more relevant attributes of a task: (yes, there are more, but not many more) task_id queue action resource state body <– The contents of this little bugger is sent to the workers result traceback error snapshot

  8. Task body { "resource": "jobs", "action": "create", "body": { "job_args": "[\"http://example.com/img.img\", \"xxxxxsha1xxxxx\"], "job_name": "test_job", "job_id": "5ed1361fd3bb4101a7457011a6e5a4d7" } }

  9. JobRunner dispatcher defdispatch(message): # Must return a tuple in the form of (result,state,error,strace) msg_body = json.loads(message.body) log.debug("Got message. Processing: %s"% msg_body) resource = msg_body['resource'] mod = importlib.import_module( 'jrunner.jobworker.callbacks.%s'%str(resource)) ret = mod.process(msg_body) # Send reply here rsp = build_response(*ret) send_reply(rsp, message) returnTrue

  10. Taskset layout

  11. https://github.com/gabriel-samfira/jrunner

  12. Questions?

More Related