Driving Quality (IV&V, QA)

1. Driving Quality (IV&V, QA) American Public Human Service Association (APHSA) IT Solutions Management (ISM) Conference Boston, Massachusetts August 7, 2007 Edward Addy, PhD, PMP Program Manager Northrop Grumman Corporation

2. QA V&V Driving Quality • Quality – Built In, Not Tested In • Two Processes that Aid in Building Quality Into the System • Quality Assurance (QA) • Software Verification & Validation (V&V) • What are these Processes and How are they Different? • …and what happened to the “I” in front of “V&V?”

3. Definitions - (From IEEE/EIA 12207.0-1996 Standard for Information Technology — Software life cycle processes) • Quality Assurance - Process for Providing Assurance that the Software Products and Processes in the Project Life Cycle Conform to their Specified Requirements and Adhere to their Established Plans • Verification – Process for Determining whether the Software Products of an Activity Fulfill the Requirements or Conditions Imposed on them in the Previous Activities • Validation - Process for Determining whether the Requirements and the Final, As-Built System or Software Product Fulfills its Specific Intended Use

4. Primary Lifecycle Processes Supporting Lifecycle Processes Acquisition Documentation Configuration Management Supply QualityAssurance Operation Development Verification Validation JointReview Maintenance Audit Problem Resolution Organizational Lifecycle Processes Management Infrastructure Improvement Training Context of QA and V&V within Lifecycle Processes (IEEE/EIA Std 12207.0) “Joint Reviews, Audits, Verification, and Validation may be used as techniques of Quality Assurance.”

5. As a Sound Bite … As an Analogy… • Quality Assurance is analogous to a code inspection during house construction • V&V is analogous to the owner hiring an independent inspector during construction • Quality Assurance - Assurance that the software products and processes are in conformance • Verification – Assurance that the software is built correctly • Validation – Assurance that the correct software is built

6. The “I” Before “V&V” • The IEEE standards allow for V&V activities to be performed by anyone, but describe a concept of independence • IEEE/EIA 12207.0-1996 Standard for Information Technology — Software life cycle processes • This process may be executed with varying degrees of independence. The degree of independence may range from the same person or different person in the same organization to a person in a different organization with varying degrees of separation. In the case where the process is executed by an organization independent of the supplier, developer, operator, or maintainer, it is called Independent Verification Process. • IEEE Std 1012-2004 Software Verification and Validation • Independent verification and validation (IV&V): V&V performed by an organization that is technically, managerially, and financially independent of the development organization. • Standard requires the specification of the degree of independence required, but does not mandate any specific degree.

7. Who Performs QA Activities? • IEEE Std 730 Software Quality Assurance Plans requires that the QA Plan identify the specific organizational element that is responsible for performing each task • The SQAP standard does not impose any restrictions on who may perform a given task • “The amount of organizational freedom and objectivity to evaluate and monitor the quality of the software, and to verify problem resolutions, shall be clearly described and documented.” • IEEE/EIA 12207.0-1996 Standard for Information Technology - Software life cycle processes requires QA to “to have organizational freedom and authority from persons directly responsible for developing the software product or executing the process” • QA activities may be performed by project personnel, staff personnel outside the project or by contracted personnel • A project may have staff dedicated to performing QA or QA activities may be performed by staff members who have additional duties

8. Primary Lifecycle Processes Supporting Lifecycle Processes Acquisition Documentation Configuration Management Supply QualityAssurance Operation Development Verification Validation JointReview Maintenance Audit Problem Resolution Organizational Lifecycle Processes Management Infrastructure Improvement Training Coverage of Lifecycle Processes by QA and V&V QA is Directed at the Project V&V is Directed at the System

9. A major goal of both QA and V&V is to identify problems and potential improvements early, when the cost of change is lower. Don’t wait until the end of the project to try to improve the quality! Coverage across the System Development Life Cycle • 1) Process implementation • 2) System requirements analysis • 3) System architectural design • 4) Software requirements analysis • 5) Software architectural design • 6) Software detailed design • 7) Software coding and testing • 8) Software integration • 9) Software qualification testing • 10) System integration • 11) System qualification testing • 12) Software installation • 13) Software acceptance support • QA and V&V are applied across all phases of the SDLC • V&V (particularly IV&V) may also perform testing in addition to that performed by the developing organization • Level of effort and skills required will vary for both QA and V&V over the course of the life cycle

10. What about the Processes and Products Not Covered? • The combination of standards-based QA and V&V still leaves a gap in assurance for correctness: • Primary Processes • Supporting Processes • Organizational Processes • Supporting Products • Some coverage of documentation, requirements configuration management, audits • Organizational Products • Some coverage of training • This coverage gap (or parts of it) is often requested under solicitation titles such as IV&V, QA, or General IT Support • If the acquiring organization wants to acquire services to cover these gaps, the services should be clearly defined in the solicitation (i.e., imposing the IEEE QA and Software V&V Standards is not sufficient)

11. DHHS ACF OCSE “Independent Verification & Validation Options” • The Office of Child Support Enforcement in DHHS ACF provides guidance on IV&V of CSE systems in a document entitled Independent Verification and Validation Options (see http://www.acf.dhhs.gov/ • programs/cse/stsys/dsts_ivv_overview.html). • “Options” refers to areas that might be evaluated, and the areas are organized into three Parts: • Part I – Planning and Management • Evaluation of project management issues and activities • Examples: Project Initiation, Business Process Reengineering, Project Planning and Reporting • Part II – Project-Wide Process • Evaluation of activities that have project-wide scope • Examples: Training and Documentation, QA, Configuration Management • Part III - Environments, Phases, and Products • Evaluations of a specific phase, environment or product of the development process • Examples: Systems Engineering, Operating Environment, Database Management

12. Comparison of IEEE 1012 to OCSE IV&V Options • Under IEEE Std 1012-2004, the V&V tasks to be performed are not usually known at the time that V&V services are acquired, while the IV&V tasks are known at the time of acquisition using the OCSE IV&V Options approach.

13. So what do I get for all these letters, and how are they different? V&V QA IV&V

14. Skills Needed for Performance • QA • QA Process (including Measurement to support Improvement Process) • Management Methodology • Development Methodology • V&V • V&V Processes • Technical Solution • Extended V&V (V&V plus coverage of the “gaps”) • V&V Process • Management Methodology, Development Methodology and Technical Solution • IV&V provides increased assurance through activities accomplished by an unbiased performer

15. Comparison Summary

16. QA V&V Questions/Comments?

17. Insert photo here IV&V and QA – Observations from Experience Maureen Finmand EDS

18. What Is It? Not always easy to understand & often used interchangeably, but bottom line intended to determine if “my requirements were met” • IEEE/EIA 12207.0-1996 Standard for Information definitions: • TechnologyQuality Assurance – process for providing assurance that the software products and processes conform to specified requirements and adhere to established plans • Verification – process for determining whether the software products of an activity fulfill the requirements or conditions imposed on them in the previous activities • Validation - process for determining whether the requirements and the final, system or software product fulfills its specific intended use • Other definitions include: • “conformance to user requirements” • “achieving excellent levels of fitness for use” • “market-driven quality” which is based on achieving total customer satisfaction

19. Why Do It? • Ensure project success • Identify software and project process issues early and take corrective action • Protect investments - getting what you paid for • Decrease delays and cost overruns • Meet funding source requirements • State legislatures and executives are requiring QA and IV&V as part of approving large-scale projects – expect an independent view on what is happening • Federal funding sources specify that there must be independent oversight as part of approving matching funds

20. Who Is Doing What? State QA IV&V S I

21. State The State agency has a quality role and responsibility • Provides quality through: • Executive sponsorship and oversight • Project management • Deliverable review and approval • Targeted quality assurance reviews • Limitations • May fail to assume or minimize “quality” role since it has been “purchased” • Difficult to find and dedicate sufficient qualified staff • Not fully independent • Focus may be limited to just a few specific deliverables • Assigned staff may not have positional authority to look critically at the agency’s performance

22. Systems Integrator Are required to have internal quality assurance • Provides quality through: • Project Management • Risk and issue identification and mediation • Change control procedures and administration • Check points at key milestones; tasks incorporated into the project plan • Best Practices and Methodologies • Executive oversight of project team • Government Business Change Lifecycle • System Development Lifecycle • Formal Quality Assurance Reviews • Reviews to determine if methodologies were followed • Review of all deliverables prior to submission • Quality Assurance • Limitations • Not a external third party

23. Who Is Doing What?, continued QA or IV&V vendor … quality is their only responsibility and can include a wide range of quality management processes… • Software Quality Assurance • SQA processes provide assurance that the software products and processes in the project life cycle conform to their specified requirements by planning, enacting, and performing a set of activities to provide adequate confidence that quality is being built into the software. • Software Verification & Validation • A disciplined approach to assessing software products throughout the product life cycle. • Strives to ensure that quality is built into the software and that the software satisfies user requirements

24. QA and IV&V QA and IV&V vendor quality management processes… • Reviews and Audits • Management Reviews - to monitor progress, determine the status of plans and schedules, confirm requirements and their system allocation, or evaluate the effectiveness of management approaches used to achieve fitness for purpose • Technical Reviews - is to evaluate a software product to determine its suitability for its intended use • Inspections - to detect and identify software product anomalies • Walk-through - to evaluate a software product. A walk-through may be conducted for the purpose of educating an audience regarding a software product. • Audits - provide an independent evaluation of the conformance of software products and processes to applicable regulations, standards, guidelines, plans, and procedures

25. Factors that Contribute to Project Failures • Contributing Factors: • Lack of senior management involvement and commitment • Lack of governance and role definition • Poor project management • Unclear contracts and weak contract management • Underestimation of resources needed – buyer and vendor • Failure to define and document requirements (functional and non-functional) • Failure to define expected deliverables including targets for quality review

26. Achieving Success • Start with the Request for Proposals • Roles and responsibilities • Identify primary targets for review – product and/or project processes • Define expected quality management processes to be used; e.g., verification, validation, audits, etc • Experience • Completed projects of similar scope and size • Performed the same types of reviews • Provide key staff with the “right” skills and experience • Adept with proven methodologies, tools and techniques • Incentives • Contribute to project success

27. Achieving Success Once Project Starts… • Governance Model • Establish reporting line to high level executive not to the PMO • Maintain independence of this team • Project planning • Allocate time and resource into the schedule • Align verification and Validation at appropriate intervals • Shared goals and outcomes

28. QA V&V Questions/Comments?