Scaling Continuous Integration

1. Scaling Continuous Integration AccuRev, Inc. 10 Maguire Road Lexington, MA 02421

2. Introduction to Today’s Speakers Chris Lucca, Technical Evangelist, AccuRev About the Speaker: Chris is a methodology and process improvement expert helping companies to discover and implement their ideal process.Chris' experience dates back to his days as a build / release specialist and software engineer for IBM-Rational. He has also filled the roles of developer, release engineer and SCM consultant for numerous organizations. In addition Chris is a certified scrum master and a frequent speaker at the AccuRev seminar series.Now Chris brings the experience and knowledge he has acquired to help keep others in the industry up-to-date on the latest developments, and how they can impact them directly.

3. Software Projects: Yesterday vs. Today • Rapidly changing requirements • Slow to respond to customer/market demand • Shorter lifecycles, requiring faster time to market • Months for a new release versus years • Extremely difficult to accurately predict • Typically underestimate by a factor of 2 • Quality challenge increasing • You might have low defects, but do you meet customer expectations? • Highly distributed teams • All local versus support for on-shore and off-shore teams

4. Why Is Process Important? • Process connects business, practices and tools • Mature processes are predictable and repeatable • Practices are more complex than ever with Agile, Scrum, CI, etc… • CI helps enable process and feedback Program and Project Management Technical Practices Supporting Tools

5. Agenda • Intro to CI • Overview of Parallel Development • Scaling CI and the Parallel Dev Challenge • Q&A

6. Agenda • Intro to CI • Overview of Parallel Development • Scaling CI and the Parallel Dev Challenge • Q&A

7. Current State of CI • It’s the industry standard • Everyone is doing it • Builds are running, but why? • Many are using CI, without knowing the value • Deploy / Automated tests are becoming part of the process • Teams are struggling to do it well • What’s important when scaling CI? • Feedback / Quality • Developer downtime • Automation

8. Traditional Development Specify Design Plan Code Integrate Write tests Test Doc Aug Sep Jan Feb Mar Apr May Jun Jul

9. Traditionally Devs Work in Isolation

10. Too Much Isolation Leads to Merge Hell

11. CI: Minimizing Integration Pain • Committing early and often is less painful • But, If we commit frequently blindly, we have a broken code base • Implied condition: each integration will not decrease the code quality • This is the key to CI • The rest is implementation details

12. CI: The Simple Implementation • How do we determine code quality? We test it • How do we test code? We build it and run tests against it • This is how we get from CI to CI servers and software builds

13. So Simple But Very Important • A build & unit test is a rudimentary check • Gives important feedback • The heartbeat of the dev organization

14. What’s CI All About? • Running builds all the time • Binaries on demand • A server that sits in the corner • Something only the release engineer cares about • The automation of the build so that developers don’t have to do it

15. That’s What CI Is All About • It’s not about running builds all the time • It’s a feedback loop, to improve quality Image Source:Wikipedia

16. CI at Scale? • With too many committers into the same code line, the likelihood something is broken right now is high • We stop caring about failures • The feedback is meaningless, and productivity suffers. CI becomes “constantly broken” • Continuous noise!!

17. Agenda • Intro to CI • Overview of Parallel Development • Scaling CI and the Parallel Dev Challenge • Q&A

18. Reasons People Work In Parallel • It’s often a business reason…not for optimization • Maintenance Release – Patches • Customer Specials - Just for one customer • Development branches – Developer spikes for experiments that will integrate at an unknown time • Code Configurations – Different configurations of code based on environment, such as UAT, QA • Special Content – Images, binaries

19. Parallel Development Supports a Complex Process • Teams want to deliver faster and receive rapid feedback • Agile, Waterfall, or any mix • Coordination of workflow and process • Mainline branching • Release 1.x, 2.x, 3.x • Customer specific codebases • Beta releases • Hotfixes • Development • Staggered features and schedules • Patches

20. Agenda • Intro to CI • Overview of Parallel Development • Scaling CI and the Parallel Dev Challenge • Q&A

21. Isolation / Integration Balance • Balance between just enough isolation and integration

22. Integration Is Easy With a Few People Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4

23. It’s Hard at Scale Chapter 1-2 Chapter 1-2 Chapter 2-3 Chapter 1-3 Chapter 1-4

24. Cost of Integration Per week: • # of check-ins per developer? • # of developers? • % of check-ins that break things? • # of problems encountered per developer? • Length of time per problem encountered? 200 devs, 2 commits each, 1% are bad = 4 breaks/day If each breakage cuts productivity 50% for 15 minutes, we lose 100 dev hours / day (@$50/hr = $5000/day) $5000 / Day = $100,000 per month $1.2 Million Per Year

25. Multi-Stage Continuous Integration ws Team A ws ws CI int CI Team B ws ws CI ws Minutes Hours Daily

26. Continuous Integration • Automated Tests • Unit Tests • Binary Deploy • Database Deploy • Web Deploy • Database Deploy • Web Deploy • Binary Deploy • IT Sign-Off • Rollback (if needed) • Database Deploy • Web Deploy • Binary Deploy • Customer Sign-Off • Continuous Integration x1 x3 x30 x60 x100s x10 x1 x4 x10 x5 x2 x1 2:00 5:00 3:00 0:10 www.synerzip.com

27. Continuous Integration • Automated Tests • Unit Tests • Binary Deploy • Database Deploy • Web Deploy • Database Deploy • Web Deploy • Binary Deploy • IT Sign-Off • Rollback (if needed) • Database Deploy • Web Deploy • Binary Deploy • Customer Sign-Off • Team Specific CI Automated Merges www.synerzip.com

28. CI for a New Code Line Must Be Cheap • Build and deployment configuration • Maximize reuse • Potentially create dynamically • Tests • Core regression suites shared across code lines • Tests for new functionality shared as code is shared • Test environments • No excuse for a lack of test environments today • Deploy “what’s in production” as a baseline

29. In Closing • Compile is the first test • Unit test, deploy, automation can come after • Get feedback as fast as possible • Different teams need different types of builds • Separate code configurations based on process • It’s about people, but tooling helps

30. SCM is More Than Just Version Control • Profound effect on the day-to-day life of developers • It’s the “hub” of the development team • Takes care of the “crown jewels” of your development process • It’s where innovation takes place

31. Best Practices For: Agile Compliance Parallel Change Management Global Visual Process Automation What Is AccuRev? Much more than Version Control… AccuRev enables Software Process Automation Version Control SCM

32. Q & A Time… Call: 1-800-383-8170 or +1-781-861-8700 Email: sales@accurev.com Visit: www.accurev.com Learn: blog.accurev.comwww.accurev.com/scm-white-papers.htm

33. Questions? www.synerzip.com Hemant Elhence hemant@synerzip.com 469.322.0349 34 34 34

34. Synerzip in a Nut-shell • Software product development partner for small/mid-sized technology companies • Exclusive focus on small/mid-sized technology companies, typically venture-backed companies in growth phase • By definition, all Synerzip work is the IP of its respective clients • Deep experience in full SDLC – design, dev, QA/testing, deployment • Dedicated team of high caliber software professionals for each client • Seamlessly extends client’s local team, offering full transparency • Stable teams with very low turn-over • NOT just “staff augmentation”, but provide full mgmt support • Actually reduces risk of development/delivery • Experienced team - uses appropriate level of engineering discipline • Practices Agile development – responsive, yet disciplined • Reduces cost – dual-shore team, 50% cost advantage • Offers long term flexibility – allows (facilitates) taking offshore team captive – aka “BOT” option

35. Our Clients

36. Thanks! www.synerzip.com Hemant Elhence hemant@synerzip.com 469.322.0349 Call Us for a Free Consultation! 37 37 37