Visual Studio ALM: Lessons Learned Through Dogfooding

1. Required Slide SESSION CODE: DPR206 Visual Studio ALM: Lessons Learned Through Dogfooding Brian Harry Technical Fellow Microsoft Corporation

2. AgendaLessons Learned Through Dogfooding • Overview • TFS at Microsoft • Getting started on a release • Development • Tracking and Reporting • Validation

3. Visual Studio 2010

4. Visual Studio ALM Progression • V1 - Transparency • V2 - Bringing together development & test • V3 - ?

5. TFS at Microsoft

6. TFS at Microsoft

7. TFS in Developer Division • 3,839 Recent users • 18.7TB Storage • 826,234,724 Files • 1,594,094 Checkins • 839,595 Work Items • At least 10 different locations, including 4 “major” ones

8. DevDiv TFS Topology

9. AgendaLessons Learned Through Dogfooding • Getting started on a release • Planning

10. Planning And Tracking The Challenge • Is the organization aligned? • How are we doing against biz objectives? • Are we making progress on the right things? What We’ve Learned • Align the organization around key objectives • Leverage TFS to reflect the planning process • Use TFS Office integration for daily tracking • Use TFS for end to end traceability and reporting

11. Our Process – VS 2010 • Main objectives Pillars • Value proposition • Value Props • Value Props • Feature Groups Feature Groups Feature Groups Feature Groups • Features Deliverables Deliverables Deliverables Deliverables

12. Value Propositions • Deliverables

13. Features • Deliverables

14. Tracking Tasks In Microsoft Excel

15. Tracking Tasks In Microsoft Project

16. Iteration Reviews – Business

17. AgendaLessons Learned Through Dogfooding • Getting started on a release • Development • Organizing the Work • Branching

18. Organizing the Work The Challenge • Incredibly difficult to drive a release with 2,000 people. High degree of churn. Complex interdependencies. Inexorable schedule The Solution • “Feature Crews” • Mini-development organization • Semi-autonomous • Accountable for delivering quality

19. Checkpoints Keep Focus • Checkpoint 0 – Scope • Checkpoint 1 – Plan • Checkpoint 2 – Demo • Quality gates & Merge

20. Establishing Architecture

21. SCRUM Is Very Popular

22. Quality as You Go • Development and test are simultaneous • Developers & Testers share responsibility • Developers & Testers share tools • Test planning • Unit testing • Functional testing • Manual testing • Performance testing • Stress testing

23. Test Planning

24. Unit Testing

25. Manual Testing

26. Performance & Stress Testing

27. Dashboards Provide Visibility

28. Builds • Every checkin built & verified • Build • Code Analysis • Checkin tests • Nightly builds provide QA updates

29. Quality Gates Prevent Chaos • Carry no debt in feature development • Feature Must Pass “Quality Gates” on Finish before • Feature complete • Test Complete • All Bugs Fixed • Security Plan • Static Code Analysis • Code Coverage • No Performance Regressions • Localization Testing • API Reviews • Feature may merge to active branch only when Quality Gates are met

30. 2005 Debt vs. 2008 Debt VS 2005 Beta 1 Product Bugs only VS 2008 Beta 1 ALL bug debt

32. Branch Strategy The Challenge • Massive division – 2,000+ people • Code churning at every level The Solution • Isolation and Integration • Isolate major feature areas in branches • Automate branch quality checks • Rhythm of regular merges • Handles integration complexity

33. Source/Branch Structure • 1 branch per product • Gated checkin procedure • VBLs build in parallel • Nightly test suites verify branch health • Regular schedule for merges • Stringent criteria for merges to main $/DEV10/Main $/DEV10/PU/ <BranchName> $/DEV10/PU /<BranchName> $/DEV10/feature/ <BranchName> $/DEV10/feature/ <BranchName>

34. AgendaLessons Learned Through Dogfooding • Getting started on a release • Development • Tracking and Reporting • Trustworthy Transparency • Top Tracking Metrics

35. Trustworthy Transparency The Challenge Each team needs to optimize process differently for their needs Leadership teams need central rollup to trackprogress. The Solution • Fantastic hub-spoke federated model… • All teams use TFS for central tracking • Each team uses best process for its needs • Single reporting system

36. Iteration Reviews – Business

37. Iteration Reviews – Business

38. Iteration Reviews – Product Unit

39. Iteration Reviews – Product Unit

40. AgendaLessons Learned Through Dogfooding • Getting started on a release • Development • Tracking and Reporting • Trustworthy Transparency • Top Tracking Metrics • Validation

41. Validation The Challenge • How do you make sure a public release is high quality? The Solution • Integration testing • Overall Gut Feel “OGF” • Stress testing • Slowdown the churn

42. Shutdown Process • Full test pass • Zero Bug Bounce (ZBB) • Zero Resolved Bounce (ZRB) • Tell Mode • Verification test pass • Ask Mode • Escrow

43. Tracking Bugs – Aggregate

44. Tracking Bugs – Rates

45. Tracking Bugs – By Feature Area

46. Tracking Bugs – By Person

47. Exit Criteria Summary

48. Summary • We have rolled our learnings back into our tools • We have encountered problems • We have learned • We have rolled our learnings back into the tool • We Believe In These Tools • Scales to Massive Commercial Development • Widely tested • I hope you can learn and benefit from our experience

49. Required Slide Track PMs will supply the content for this slide, which will be inserted during the final scrub. Track Resources • Resource 1 • Resource 2 • Resource 3 • Resource 4

50. Required Slide Resources Learning • Sessions On-Demand & Community • Microsoft Certification & Training Resources www.microsoft.com/teched www.microsoft.com/learning • Resources for IT Professionals • Resources for Developers • http://microsoft.com/technet • http://microsoft.com/msdn