1 / 17

Jenkins Scalability Summit

Jenkins Scalability Summit. Oct 2013. Logistics. Where is …? Network. Agenda Part 1: Story time. Learn from our collective experience Identify high-priority problems What is hurting serious users? Concrete details are good. Agenda Part 2: Discuss & Design.

fabian
Download Presentation

Jenkins Scalability Summit

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Jenkins Scalability Summit Oct 2013

  2. Logistics • Where is …? • Network

  3. Agenda Part 1: Story time • Learn from our collective experience • Identify high-priority problems • What is hurting serious users? • Concrete details are good

  4. Agenda Part 2: Discuss & Design • Collectively pick a few topics • Split into 2 tracks • See if we can start shaping up solutions • Please keep shared notes • Details!

  5. Possible topics includes … • CPU/memory/disk/network/… consumption • Stability/diagnosability of slaves • Access control of builds/UI • Organizing jobs and build records • Master to master communication • Workflow / choreography • User interface • Stability of Jenkins releases • Plugin compatibilities

  6. Goals • No one goes listen-only mode • Produce notes to show to the broader community • Figure out how to do this better next year

  7. What We’ve Done&What We Can

  8. Thread Reduction • Per-slave consumption • SSH (down to 1) • Channel (down to 1) • Executor thread on demand

  9. Thread Reduction: TODO • NIO • Asynchronous job execution • More about this in workflow

  10. Memory reduction • Lazy loading of build records • Database plugin

  11. Design Choices That Constrain Us • Compatibility • Adding is doable, changing is often hard • Heterogeneousness in data model • Thread-driven execution model

  12. CloudBees Scalability Efforts

  13. Multi-master / Meta Jenkins • Master-to-master channel • Extensions on top of it • “Cloud” implto lease slaves • Push security realm • Push update center • …

  14. Multi-master • Where we think we can go with this • Move jobs around • Sharding with UI mashup • Where this wouldn’t take us • Loss of inflight builds • True horizontal scaling

  15. Workflow • Brand new job type • Kill multiple birds in one stone • Inspired by buildflow & jenkow

  16. Ingredients • No single thread to execute the whole thing • Analogous to NIO • Check pointing • BPMN-like workflow execution model • Surface syntax independent • Groovy DSL • With continuation-passing style execution model • Post-execution visualization • But no pre-execution visualization

  17. Ingredients • Unified properties, build variables, and environments • Open up direct access to file store • Hands-free process forking • Interop with existing job types

More Related