Implementing Zero-Debt Continuous Inspection in an Agile Manner

1. ImplementingZero-Debt Continuous Inspection in an Agile Manner A Case Study Brian Chaplin October 8, 2013

2. Agenda • What is technical debt? • What is continuous inspection? • A scalable framework to automate technical debt reporting • How to control technical debt using the framework • Best practices • Lessons learned from 2 case studies • Agile lessons learned • What you can do tomorrow

3. Technical Debt Metaphor • Ward Cunningham at OOPSLA 92 : “Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite...” • However, • Every minute spent on debt counts as interest

4. Consequences of Technical Debt • Product atrophy • Decreased predictability • Underperformance • Universal frustration • Decreased customer satisfaction • Unpredictable tipping point • Increased time to delivery • Significant number of defects • Rising development and support costs From Essential Scrum, Kenneth S. Rubin

5. Types of Debt • Naïve • reckless debt (Fowler 2009), unintentional debt (McConnell 2007), and mess (Martin 2008). • Unavoidable • Although this debt might be predictable it’s not preventable because we can’t foresee how the component might evolve • Strategic • deliberately make a strategic decision to take shortcuts during product development to achieve an important short-term goal, such as getting a time-sensitive product into the marketplace.

6. Naïve Technical Debt Defined

7. What is Continuous Inspection • an approach to running automated code analysis as part of a build in order to find code quality problems

8. Why Continuous Inspection? • 5. Sustainable development, able to maintain a constant pace • 9. Continuous attention to technical excellence and good design • can help reduce the time spent in long, manual code review sessions

9. Continuous Inspection Tooling • SonarQubewww.sonarqube.org • CAST www.castsoftware.com • Microsoft Visual Studio/Team Foundation Server

10. Automating Continuous Inspection Reporting • Necessary for large projects • Agile and timely reporting within hours • Necessary for zero-debt • Architects only have time for the big issues • Computer handles the smaller quality defects

11. Business Context • Large project, semi-Agile • IT history of past quality initiatives • OO and pair programming • Automated unit test generation • Management goal of 80% test coverage • Continuous integration build/deploy

12. Characteristics of 2 Case Studies

13. Architecture Requirements for Code Quality (CQ) Statistics • Must come from existing continuous integration process • Available within 24 hours • Meaningful to developers, leads and management • Must involve reviewers • Must tie to accountability points • Leads, reviewers, submitters, project, business function

14. Architecture ETL Overview • Extract, transform and load • Match the committer with the quality change

15. Architecture Overview • Extract/Transform/Load (ETL) • Input • Source code management system (SCM) • Project management system • Code quality metrics • Output • SQL database • Excel pivot table reports • Defect tickets (project management system) • Email • Ad hoc management reports • Program scores and developer contribution

16. Establish and Maintain the Quality Database • Guard against corrupted quality data • Test or build failures • Sonar may record lower stats • Rollback a bad build • Design for mash-ups • Combine code quality with runtime defect reporting

17. Maximize Report Flexibility • Enables timely and actionable reporting • Excel pivot table as data-marts • Refreshed Excel reports accessed via Email hyperlinks or served up by web server

18. Fast Feedback Bake Code Quality into the Build and Sprint Notificationsin every phase Sprint Code Quality ChangeTechnical DebtOpen Tickets To Fix Debt DailyIntegration Tests Code Quality Champions Personalized CQ Report ContinuousAuto Build Unit TestDeploy Email Code Quality Warnings

19. Technical Debt Information Radiator • Fast feedback: identify and notify Immediately • Inside the IDE • After the commit • Track until it’s fixed

20. Fast Feedback 1. Notify Immediately • Email each debt violation within hours • After each continuous inspection build • 3 times a day or after every CI build • Notify both committer and code reviewer • Daily contribution • Best contributors summary • Personalized contribution detail to each committer • Hyperlink to more reports and wiki • Direct Sonar hyperlink to the degraded file

21. Submission Warning(s) You are receiving this message because even though the code quality may have been enhanced, the submission(s) below have decreased the code quality (coverage, compliance, and/or uncovered complexity) or they didn't meet the standards for a new class. Please review these submissions with your reviewer and take the appropriate action. Sample Violation Email

22. 2. Track Until It’s Fixed • Metric • Submitter • Lead • Reviewer • Class • Sprint • Date

23. 2. Track Debt Until It’s Fixed

24. Making Technical Debt Visible • Estimate the cost of remediation • Assign owner • Report weekly progress by owner • Report trends • Monetize it, various methods • SQALE (Software Quality Assessment based on Lifecycle Expectations) from Inspearit, Jean-Louis Letouzey

25. Monetize the DebtEstimated Remediation Costs

26. Report Weekly Progress by Owner Ratio of bad to good code Weekly progress Current debt

27. Report Weekly Tech Debt Trends • By project • Or by owner

28. Best Practices • Staff a code quality desk • Knowledge clearinghouse • Use code reviewers • Stabilize build and project structure • Establish static code analysis rules and rarely change them

29. Best Practices, cont’d • Developers must be able to clear unfixable debt from the tracking report • Run the ETL at least daily • Keep the database accurate • Track both contribution and debt • Recognize code quality champions • Use uncovered line/branch count not percentage

30. Allow for Exclusions, False Positives • Integrations, class rename, code moves • Classes exempted from quality • Registries • Test support • Unfixable debt • Unreachable test cases • Exceptions to the metrics standard • Static analysis false positives • 0.5% are excluded

31. Case Study Lessons Learned • Communicate the benefits of code quality • Maintainability • Less runtime defects, explain carefully • Working with a net • Help new developers with un-testable code • Train and mentor • Consider refactoring to testable code first, then write the test

32. Reviewers Guard Against Doing the Wrong Thing • Meaningless comments • Unit tests just for coverage that assert nothing • Allow some large case statements

33. Too Busy to Keep Debt Down?

34. Managing the Debt Rate • Debt rate is about 15% for both projects • Varies widely by developer Monthly debt rate

35. Techniques to Keep the Debt Rate Low • Immediate feedback • Open defect tickets at key points • Train • Assist developers one on one • Encourage re-factoring to write testable code • Code a little, test a little • Keep management aware • Continually monitor, don’t let it get out of hand • Recognize the champions

36. Does bad code cause defects? 10/20 top NPEs predicted Probability =exp( a + b*WEIGHTED_VIOLATIONS + c*COMPLEXITY -d*COVERAGE - e*COMMENT_LINES_DENSITY - f*SCORE)

37. Implement the “Boy Scout rule” • Encourage Code Quality Improvement On Every Commit • Score the program’s improvement • Score the committer’s improvement • Recognize the contributors • Ensure the numbers are used properly

38. Agile Lessons Learned

39. When and How to Create Debt Backlog Items • Detailed appropriately • By metric, program file, developer • Emergent • At release code branch • Move to later Sprints as necessary • Estimated • Use Sonar tech debt estimate • Prioritize by • Complexity • Unit tests • Static code

40. Keep the Pressure Off • Be positive! • Manage debt informally as sprint issues • Use TDD and other techniques as productivity aids • Interpret the contribution metrics correctly

41. Fast Feedback • Reviewers work with submitters to fix same day • Quick emails with links to more information • Allow developers to correct immediately after receiving an email

42. Small Batch Size • Fix the debt as it’s incurred • Don’t let it build up • Minimize tech debt inventory • Static code violations • Code and branch coverage • Complexity management • Circular dependencies • Fan-out reduction

43. Individual vs. Group Accountability • Idle work vs. idle workers • Attack the most significant debt first • Fix your own debt quickly • You know the code best • Fix debt in the code you’re working on • Leave it cleaner than you found it

44. Minimize Ceremony • Backlog spreadsheet always available • Morning “warm-up”, fix a little every day • Only open backlog tickets at release or sprint boundaries

45. Code Quality Take-away Messages

46. Quality Motivates and Enhances Productivity • Tom DeMarco & Timothy Lister, Peopleware: “Quality, …, is a means to higher productivity” • Defect removal #1 cause of productivity loss • Programmers are motivated by emphasis on code quality

47. Implement CQ Continuous Inspection • Establish CQ point of accountability • Promote CQ • Track technical debt • Require remediation • Scales to large projects • Multi-branch • Multi-language • Target the reports • End of Sprint or production release branch • By lead, committer, reviewer • Commit quality trend

48. Embed a Code Quality Expert on the Team • Unit testing techniques • Mocks • Corner cases • When to override the warnings • When to clear, not fix, debt • Safe re-factoring

49. How Much Time for a Code Quality Czar? • 12% of a senior dev’r

50. You Can Maintain Velocity and have Zero Debt Keep the code base maintainable so velocity continues 13% more code per year All with 100% test coverage