Architecture Review Boards

1. Architecture Review Boards Barry Boehm 10/14/2009

2. Outline • AT&T/Lucent ARB Concept • -Overview • -Results • -Recommendations • USC CS577 ARB Concept • -Participants • -Procedures • -Results

24. Outline • AT&T/Lucent ARB Concept • -Overview • -Results • -Recommendations • USC CS577 ARB Concept • -Participants • -Procedures • -Results

25. USC CS577 ARB Participants • Project Team • -Everybody presents something • Reviewers • -Clients • -Instructors and TA’s • -Industry participants • 80 minute time slots

26. ARB/milestones for two-semester team • FCR ARB: October 19th to 23rd • Based on preliminary FC package • Focus on FCR success criteria • DCR ARB: November 30th to December 4th • Based on draft DC package • Focus on DCR success criteria

27. ARB/milestones for one-semester team • DCR ARB: October 19th to 23rd • Based on DC package • Focus on DCR success criteria • CCD: November 11th • Core Capability Drive-through • Client(s) will have hands-on experience on your core capabilities • TRR: November 23rd • Informal review with TA; readiness for transition • OCR: November 30th to December 4th • Based on AsBuilt package

28. ARB Review Success Criteria • FCR • For at least one architecture, a system built to arch. will: • Support the Ops Concept • Satisfy the Requirements • Be faithful to the Prototype • Be buildable within the budgets and schedules in the Plan • Show viable business case • Key stakeholders committed to support Foundations (nee Architecting or Elaboration) Phase (to DCR) • DCR • For the selected architecture, a system built to the arch. will: • Support the Ops Concept • Satisfy the Requirements • Be faithful to the Prototype • Be buildable within the budgets and schedules in the Plan • All major risks resolved or covered by risk management plan • Key stakeholders committed to support full life cycle

29. ARB Review Success Criteria CCD TRR / OCR • Determine whether client needs anything further to ensure successful Transition and Operation • Changes in priorities for remaining features? • Changes being made to operational procedures? • More materials needed for training? • Changes in system data or environment we should prepare for? • Anyone else who should experience CCD? • Show value • Product works as expected (or not with noted exceptions) • Product will help users do job • Show quality development • e.g. relevant parts of your • IOC documentation • Process • Show sustainability • e.g. support requirements/plans • Transition plan & status • Show confidence that product is/will be ready to be used • e.g. Transition plan & status • See also value

30. TeamPreparation for ARB Reviews • Week-1 Within-team Dry run of presentations and demo Further dry runs as necessary • ARB Week ARB Presentation and discussion Follow-up team discussions, client discussions • Week+1 Monday: FC packages due Monday: DC packages due

31. FCR ARB Session OutlineArchitected Agile Team (x,y): (presentation time, total time) (5 , 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; desired capabilities and goals (10,10) Prototype. Most significant capabilities [buying information](especially those with high risk if gotten wrong) (5, 10) Requirements. Most significant requirements (5, 10) Architecture. Top-level physical and logical architecture; status of NDI/reuse choices (5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (5, 10) Feasibility Evidence. Business case (beginnings, including benefits analysis); NDI analysis results; major risks; (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10) Things done right; issues to address (Instructional staff) • Plan on 2 minutes per briefing chart, except title • Each chart MUST have information specific to your project

32. FCR ARB Session OutlineNDI/ NCS Team (2 semesters) (x,y): (presentation time, total time) (5 , 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5 , 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype. Most significant capabilities, NDI/NCS integration (5, 10) Architecture. Top-level physical and logical architecture; (5, 10) Life Cycle plan. Life cycle strategy; focus on Foundations phase; key stakeholder responsibilities; project plan, resource estimation (10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (10) Things done right; issues to address (Instructional staff) • Plan on 2 minutes per briefing chart, except title • Each chart MUST have information specific to your project

33. DCR ARB Session OutlineNDI/ NCS Team (1 semester) (x,y): (presentation time, total time) (5 , 5) Remote Team Member(s) (jointly) Team’s strong points & weak points (operational view and technical view) concerns & possible solutions (10,10) OCD. System purpose; shared vision; proposed new system; benefit-chain diagram; system boundary; core capabilities, constraints and goals (5 , 5) WinWin Agreements. Agreed Win conditions in each category (10,10) Prototype/ Product Demo. Most significant capabilities, NDI/NCS integration (5, 5) Architecture. Top-level physical and logical architecture; (if applicable) (5, 10) Life Cycle plan. Life cycle strategy; focus on Development phase & transition increment; key stakeholder responsibilities; project plan; resource estimation (10, 15) Feasibility Evidence. NDI/NCS alternatives, NDI/NCS evaluation & analysis results; Business case (beginnings, including benefits analysis); major risks; Capability and LOS feasibility evidence (5, 5) QFP. Traceability Matrix and summary; Defect Identification review type summary (what & how) by document section or UML, and current defect injection & removal matrix (5, 5) Acceptance Test Plan and Cases (10) Things done right; issues to address (Instructional staff) • Plan on 2 minutes per briefing chart, except title • Each chart MUST have information specific to your project

34. Team’s strong points & weak points • List at least one item for each of the following • List your team’s strong points • Operational view • Technical view • List your team’s weak points • Operational view • Technical view • Identify specific technical concerns & possible solutions • Identify operational risks & possible mitigation • Sources of observations • Team activities, package evaluation, winwin negotiation, and etc.

35. Defect Identification Review • For each document section, UML model, and etc. identify the following • type of review you used (peer review, agile artifact review, and etc. ) • Other form of defect identification, e.g., grading, client feedback, etc. • Current Defect Injection and Removal Matrix • Current, total defect information from your progress report

36. ARB Session Outline • DCRSimilar format to FCR, different focus: • Less time for OCD, Prototype • More time for Architecture, Plans • Details TBD based on FCR ARB experience • General rule on focus: emphasize your projects high risk areas • At FCR (in most cases) this will involve establishing the operational concept (including system analysis) • At DCR (in most cases) this will involve the system design and development plan (especially schedule)

37. Results To Date • * • - Poor performance • Poor team management • Poor communication (within team and with client)

38. ARB Packages • If you would like to have your ARB presentation and FC/DC package printed for you at the ARB meeting, • email the documents (in a zip file) to csci577@usc.edu24 hours in advance, unless your ARB is on Monday, in which case, email the documents 5 hours in advance. • Otherwise, bring 5 copies of your ARB presentation and 2 copies of your FC package to the meeting.

39. Demos in ARB • For those teams doing a live demo in the ARB meeting, please include screenshots of your demo in your presentation for your IV&Vers to see the demo.

40. ARB Presentation slides • Upload your ARB presentation slides (before your ARB) on your team website for off-campus students.

41. Web-ex & Teleconf • Off-campus student who can not attend the ARB in-person, will be connecting through web-ex

42. Past FCR Experiences and General Guidelines

43. Outline • FCR ARB 2008 Feedback Summary • Examples of Good and Bad Practices seen at ARBs • ARB Chartsmanship & Presentation

44. Overall FCR Feedback – Fall 2008 • Generally done well: presentations, time management, client rapport • Reconcile FCR content with pass/fail criteria • Define key terms, acronyms in glossary • If you’re deferring or skipping a normally-included artifact, explain why (e.g. COTS internals unavailable)

45. OCD Feedback – Fall 2008 • Generally done well: Organizational goals, Operational concept, System boundary and organizational environment. • Benefits chain needs rework • Added stakeholders: users, clients, developers, IV&Vers, database administrators, maintainers, interoperators, suppliers • Assumptions are about environment not about outcome • Organization Goals are Benefits Chain End Outcomes, not project Initiative contributions • Business Workflow – not technical workflow • Prototype and System are NOT the same

46. Prototype Feedback – Fall 2008 • Generally done well: GUI Prototypes, Good understanding of client’s needs • Prototype all high-risk elements, not just GUI’s • COTS interoperability, performance, scalability • Use user-friendly terms (John Doe, 22 Elm St.) not abstractions (N.,A or test test test) • Identify end users and try to get feedback from end users • Focus on important and high priority requirements (initially)

47. SSRD Feedback – Fall 2008 • Generally done well: Project and Capability requirements, OCD-SSRD traceability • Prioritize all the requirements • Propagate LOS goals from OCD into SSRD or drop LOS requirements from SSRD (and SSAD) • Distinguish between client imposed requirements (SSRD) and developer choice solution (SSAD) • Make sure all requirements are testable • Qualify “24/7” Availability with noted exceptions

48. SSAD Feedback – Fall 2008 • Generally done well: Overall views • Follow UML conventions (arrows, annotations, etc.) • Follow the SSAD guideline • Read the exit criteria for the milestone carefully

49. LCP Feedback – Fall 2008 • Generally done well: Overall strategy, roles and responsibilities • Fill in 577b TBDs • Full elaboration phase plan • COCOMO drivers differ as per the module • Add IV&Vers interactions to the Gantt Chart • Add maintainer’s responsibilities

50. FED Feedback – Fall 2008 • Generally done well: Business case framework, risk analysis • Specify LOS feasibility plans • Include training, operations, maintenance opportunity costs/ effort • Drop developers hours as cost • Try to quantify benefits, show return on investment • Change ROI to yearly vs. by semester • Distinguish one-time from annual costs in business case • Benefits start in 2007 • Requirements traceability: Replace the generic requirements traceability table. Show traceability between items not sections • Co-ordinate risk in LCP and FED • Elaborate process rationale