How Microsoft does “Software Engineering”

1. How Microsoft does“Software Engineering” Myung Ho Kim National Technology Officer (NTO) Microsoft Korea

2. Overview • Product Group Organization • Engineering Software Products • Hard Decisions and Wisdom

3. Product Group Organization Feature-set Architecture Implementation Testing Quality assurance

4. A ‘Typical’ Product Group • 25% Developers • 45% Testers • 10% Program Management • 10% User Education / Localization • 7% Marketing • 3% Overhead

5. The Developer Division (DevDev) • Has about 2,200 people • Visual Studio / .NET Framework: 30+ Product Units

6. VS 2005 & .NET Framework 2.0 • Over 8 million lines of code • Over 15,000 files • Over 150 binaries

7. Typical Project Cycle Vision statement RTM Feature complete Release Testing Planning M1 M2 Technology Preview Beta 1 Beta 2 RC1

8. Planning Phase • Marketing gathered requirements. • Senior leadership provided an initial vision. • A cross discipline team created a vision statement and elaborated on product themes. • The same team created an overall schedule. • Product Units began a detailed planning of what features would get into the release.

9. Scheduling * • Milestone = unit of product cycle progress • Enable accurate assessment of progress and distance left • Milestone features are scheduled in priority order • Enable flexible scheduling to respond to feedback later • Milestone is done when quality exit criteria is met • Forcing function to ensure team doesn’t go fast and loose • Ensure very stable, usable, and complete product at exits

10. Key Feature Milestone Events

11. Design Specs • The feature-set is driven by Program Managers • Own the customer experience and making the right thing happen • Drive the product unit team during the design process • Own writing a design specification for each feature

12. Coding • Developers use a variety of tools for coding. • Visual Studio, Emacs (huh!), vi (huh!!), Source Insight… • No tool-specific files or code-injection allowed to be checked-in • Maintain single source tree for entire product line. • Can type “build -c” from root to build from scratch • Can type “build -c” in any sub-directory of source tree • Each team maintains a private branch off main tree. • Use internal source control system optimized for branching/merging • Minimizes check-ins into main tree and avoids breaks

13. Coding • C++ & C# are the 2 common languages used. • The goal is efficient, clean, and maintainable code. • Code we ship lives at least 10 years guaranteed support • Documentation and resource strings are stored externally from source code. • Avoid documentation writers colliding with developers • Localization work handled centrally in Ireland & Japan • .NET Fx is localized into 34 languages, VS 2005 into 8

14. Coding • All code check-ins must be reviewed by another developer on the team prior to submitting any changes to the source tree. • Dev is responsible for “check-in suites” to exercise implemented features. • All product check-in suites must run and pass 100% before any check-in can be submitted into source tree (2-3 hour process) • “Check-in mail” is sent to the team summarizing changes made, bugs fixed, and files modified.

15. Producing Daily Builds * • A new “build” is built and released every day. • Forces engineering discipline, critical in the end-game • Build number indicates build date (40730: Jul. 30th) • Central build lab produces daily builds for entire division • Staffed 24 hours a day throughout the week • Build process: • Build team syncs source code tree (~mid-night) • Kick off end-to-end build (~4:00am) • Build complete (~1:00pm) • Perform BVT run to verify build works (~4:00pm) • Pick up hot-fixes & re-build for BVT failures • Repeat hot-fix/BVT cycle until the build passes clean

16. Testing • Test team is staffed by Devs • Designs test plans, writes tests, and builds infrastructure • Target goal with tests is to reach >70% product code coverage by end of the product cycle • We measure this statistic throughout the product • In Visual Studio 2005 - 62% (native) and 70% (managed) • The Visual Studio 2005 test status: • 10,339,207 functional test cases • ~9,000 servers in test lab to run them in automated way • A full test pass takes 21 days • Test management system manages test plans/runs

17. “Mad-dog” Test Automation System

18. Bug Tracking • Bugs & work-items are tracked within an internal bug-system “Product Studio”. • Enables rich reporting and change history tracking • Bugs triaged by feature leads and assigned priority • Bugs fixed in priority order of range from 0 to 4 • Daily status mails sent to entire division tracking bug status • After the final feature-milestone is completed, life of the product team revolves around driving it to zero.

19. Ship Mode: Coasting In • Large projects “coast in” r.t. ending suddenly. • You can’t dock a large ship abruptly. • Key set of steps along the “coasting path” • Lock feature-set and stop adding/changing design • Run a full test pass to find all bugs of the locked design. • Push for a zero bug bounce (ZBB) on the locked design. • Absorb bug-bounce over few weeks from ZBB. • Triage all bugs out of the system that are not “must fix”. • Enter End-Game mode & start minimizing code churn.

20. End-Game Mode • “War Team” takes over to the End-Game mode. • Made up of most senior team members, all with ship experience. • Guides the ship in into port. • Meets at least once a day,  twice a day at the very end. • Over the span of the final weeks before a release • War team steadily raise the triage bar, and towards the end only allow “showstopper” bugs to be fixed. • If product is ready to ship, the number of showstopper bugs will slowly decline as check-in rates drop.

21. End-Game Mode • Shipping is more of an art r.t. a science • Senior leaders need to feel that the product is right, have confidence in test coverage, customer scenarios… • Once the rate of incoming bugs gets to zero, put the build in escrow for a few days, and wait & see. • If it holds, ship it. • If not, repeat until we are finally there. • Milestone ends • Rinse and repeat (Iterative process!) • Visual Studio 2005 went through M0 (planning), M1, M2, Beta 1, M3.a, M3.b, M3.c, Beta 2, RTM

22. From Bugs to Builds Component triage Dev coding & Unit testing Buddy testing Code check-in War Team Daily build Bug found or escalated Build available CDs RTW Pkg BVT Dogfood

23. Customer Connections • We realized that our customers need to be part of our development. • We had to break the pattern that let us think that “we know what’s best”. • Four areas of focus: • Product Feedback Center (Ladybug) • Self-sustaining Communities (Forums) • Community Technology Previews (CTP) • Technical Assistance Program (TAP) • Refer to site http://blogs.msdn.com/ddcpxblg

24. Hard Decisions • The Golden Rule • Features + Quality = Time • You can only control two from these three! • Development focus had (always) been Q & F. • Don’t worry too much about the schedule • Front Loading Quality • Maintain a very high quality standard throughout the project to mitigate F/Q/T tradeoffs • No feature work items get checked-in until they are completely passing tests; there is always next release

25. More Wisdom of the Ages • Products ship with bugs... • The last bug is found when the last customer stops using your product • People will always screw up a good project • Don’t add them unless you really have to • “Mythical man month” • The world is not so static from specs thru RTM • Resources, priorities, markets, features… will change • Focus on what ultimately matters • Quality • Compelling to customers • Solving business problems

26. A Decade of Advances in Software Constructionand Microsoft’s Scorecard • Design has been raised a level • Daily build and Smoke test • Standard libraries • Visual programming innovation • Nice community of programmers • The Web, for research • Widespread use of incremental development • Test-first development • Refactoring as a discipline • Faster computers -- Source: Construx, Code Complete 2: Realities of Modern Software Construction, 2004

27. Software is built by People • Understand what’s really bad for morale • Long hours and high stress • Unrealistic schedules • Feeling of low productivity • Working on poor-quality software

28. Software is built by People • Watch out workplace demons! • Dishonesty • Sloppiness • Backbiting • Intolerance • Jealousy • Sexism • Carelessness • Inability to speak to power • Lack of respect for the other • Blaming • Please come out from the color of the night…

29. Closing Remark • Factors that impact any successful project • Technology • Process • People