Welcome

2. Welcome • Welcome to the Programmatic Audit and Review (PA&R) Process training module. Module I begins with a basic introduction to the elements of the Office of Safety and Mission Assurance (OSMA) Review and Assessment Division (RAD) and provides a detailed description of the Programmatic Audit & Review (PA&R) Process. • The PA&R Process directly supports the OSMA/RAD charter to ensure lifecycle safety and mission success through compliance verification of Agency SMA requirements thus providing a robust foundation for the Safety and Mission Assurance (SMA) operational readiness decisions.

3. Training Agenda • Module I - Introduction and PA&R Overview • Module II- Knowledge • Make Decision to Audit • Conduct Program Discovery • Understand Baseline SMA Requirements • Module III - Visibility • Verify Requirements Flow-down • Verify Process Capability • Verify Requirements Compliance • Provide Feedback • Module IV – Understanding • Support Operational Readiness Decision • Module V – Summary

4. Module I - Introduction & Overview • Why Conduct a PA&R? • What is a PA&R? • How Do We Implement the PA&R Process?

5. Why Conduct a PA&R? Module I “ Prepare a detailed plan for defining, establishing, transitioning, and implementing an independent Technical Engineering Authority, independent safety program, and a reorganized Space Shuttle Integration Office…” Recommendation R9.1-1 of the CAIB Report dated August 2003 The Programmatic Audit and Review (PA&R) Process was initiated in response to the Columbia Accident Investigation Board (CAIB) Findings and Recommendations to provide a robust, structured and formalized independent Safety and Mission Assurance Compliance Verification function in the Agency. It represents, in part, the Agency response to CAIB recommendation R9.1-1 and R7.5-2 on the need to increase independent assessment and compliance verification activities, and is in accordance with commitments by the Administrator to initiate Space Shuttle Program independent review and assessment activity prior to Return to Flight (RTF). “NASA Headquarters Office of Safety and Mission Assurance should have a direct line authority over the entire Space Shuttle Program safety organization and should be independently resourced.” Recommendation R7.5-2 of the CAIB Report dated August 2003 This PA&R function was established under a new Review and Assessment Division of the Office of Safety and Mission Assurance in 2003 and was chartered via the normal NPR processes.

6. OSMA Safety and Assurance Requirements Division (SARD) Corporate leadership in definition and implementation of NASA's Agency-wide Safety and Mission Assurance Policies, Procedures, Standards, Tools, Techniques and Training. Mission Support Division (MSD) Primary interface with the customers of the Office of Safety and Mission Assurance Assurance that appropriate SMA requirements are in place for each program or project. Review and Assessment Division (RAD) Primary Independent Corporate Compliance Verification Responsibility Best Practices & Lessons Learned Pre-Operations Decision Support Knowledge, Understanding & Visibility of Compliance with Requirements, Process Capability, Technical Issues, Minority Opinions & Residual Risk Office of Safety and Mission Assurance Module I The Office of Safety and Mission Assurance (OSMA), as well as Center Safety Mission Assurance (SMA) organizations, play a key role in the PA&R Process. The chart below depicts the PA&R-related responsibilities of OSMA. NASA Centers may have similar relationships. Although all divisions of OSMA have a role, RAD has the primary OSMA compliance verification responsibility.

7. Compliance Verification of Agency SMA Requirement Set Agency SMA Review & Assessment Functional Audits Agency Requirements Institutional / Facility / Operational (IFO) Safety Audit Institutional Programmatic Support (IPS) Audit Center Requirements Program Requirements Programmatic Audit & Review (PA&R) Contracts Contractor Requirements The RAD Compliance Verification Landscape Module I

8. The RAD Compliance Verification Landscape Module I Compliance Verification of Agency SMA Requirement Set Agency SMA Review & Assessment Functional Audits Agency Requirements Institutional / Facility / Operational (IFO) Safety Audit Institutional Programmatic Support (IPS) Audit Center Requirements Program Requirements IFO Safety Audits are conducted on site at the NASA Centers to verify compliance with Federal, State, local, and Agency SMA and technical directives, policies, and requirements contained in applicable documents. These requirement areas within the IFO Baseline Requirements Set (BRS) include, but are not limited to Public/Worker Safety, Facility System Safety, Mishap Investigation, GIDEP, Fire Protection, Equipment Safety, Explosives Safety, Range Safety, and Electrical Safety. Programmatic Audit & Review (PA&R) Contracts Contractor Requirements

9. The RAD Compliance Verification Landscape Module I Compliance Verification of Agency SMA Requirement Set Agency SMA Review & Assessment Functional Audits Agency Requirements Institutional / Facility / Operational (IFO) Safety Audit Institutional Programmatic Support (IPS) Audit Center Requirements IPS Audits are conducted onsite at the NASA Centers to verify that Institution/Center policies, procedures and processes are in place to define, impose, and implement appropriate Agency SMA requirements for their respective programs and projects. Program Requirements • The purpose of the IPS is twofold: • Provide management with an independent, objective, and constructive evaluation of the compliance of the institution with the applicable IPS BRS. • Assess the effectiveness of implementation of SMA process and technical directives, policies, and requirements in achieving desired outcomes relevant to the areas being audited. Programmatic Audit & Review (PA&R) Contracts Contractor Requirements

10. The RAD Compliance Verification Landscape Module I Compliance Verification of Agency SMA Requirement Set Agency SMA Review & Assessment Functional Audits Agency Requirements Institutional / Facility / Operational (IFO) Safety Audit Institutional Programmatic Support (IPS) Audit Center Requirements Program Requirements Programmatic Audit & Review (PA&R) Contracts The PA&R process provides independent compliance verification of Agency SMA process, technical, and, as appropriate, engineering performance specification (EPS) requirements within the applicable program/project BRS. Contractor Requirements PA&Rs include analyses, reviews, and assessments in addition to the on site compliance verification audits at center, prime contractor, and subcontractor facilities as appropriate

11. Compliance Verification of Agency SMA Requirement Set Agency SMA Review & Assessment Functional Audits Agency Requirements Institutional / Facility / Operational (IFO) Safety Audit Institutional Programmatic Support (IPS) Audit Center Requirements Program Requirements Programmatic Audit & Review (PA&R) Contracts Contractor Requirements The difference in RAD programmatic audits is that IPS Audits focus on Center SMA support and the imposition of requirements on Center programs and projects, while PA&Rs focus on specific and unique program and project SMA functions and the implementation of requirements. The RAD Compliance Verification Landscape Module I

12. Authorizing Documents Module I There are several Agency level documents that authorize compliance verification and the PA&R Process: NASA Policy Directive NPD 1000.0, Section 3.2 states that “NASA employs a system of checks and balances for effective internal control and to ensure the successful achievement of missions, assigning proper levels of influence and action to different organizations. … The Safety & Mission Assurance organization maintains responsibility for verification of programmatic compliance through strategies, policies, and standards.” NASA Policy Directive NPD 1000.3C, Section 4.6.2.2 (d) states that “the Chief, Safety and Mission Assurance is responsible for …(d) Verifying the effectiveness of SMA requirements, activities, and processes.” NPR 7120.5D, paragraph 2.5.4 states that “NASA HQ SMA also has a Programmatic Audit and Review (PA&R) process described in NPR 8705.6, Safety and Mission Assurance Audits, Reviews, and Assessments. That process provides independent compliance verification for the applicable NASA SMA process and technical requirements within the program/project safety and mission assurance plan, the program baseline requirements set, and appropriate contract documentation. Program/Project Managers directly support the PA&R process (either Headquarters-led or Center-led) by providing logistics and resource support required for the successful execution of and response to PA&R process activities. They also coordinate with Center SMA and Center procurement officials to ensure that contracts provide for adequate contractor support for all PA&R activities, and they direct and authorize program/project contractors to support PA&R process activities.” NPD 8700.1C, Section 1.e states that “it is NASA policy to verify and validate life-cycle implementation of SMA, RM, and mission- success requirements through ongoing surveillance of program, project, and contractor processes.”

13. Office of Safety and Mission Assurance Headquarters Office Work Instruction HOWI 8700-GE037 Baseline January 17, 2006 NASA PROCEDURAL REQUIREMENTS NPR: 8705.6 Effective Date: October 04, 2005 Expiration Date: October 04, 2010 Programmatic Audit and Review (PA&R) Process Safety and Mission Assurance Audits, Reviews, and Assessments • 1. Purpose • The purpose of this Office of Safety and Mission Assurance (OSMA) Headquarters Office Work Instruction (HOWI) is to document the process of preparing for and conducting Programmatic Audits and Reviews( PA&R) of NASA programs and projects. This HOWI is an integral part of the NASA Safety and Mission Assurance (SMA) Review and Assessment process, which also includes the NASA Headquarters Institutional/Facility/Operations (IFO) Safety Audit process and the Safety and Mission Success Review (SMSR) process, discussed in their respective HOWIs. Responsible Office: Office of Safety and Mission Assurance • Section 3.2.7.1 [Program/Project Manager shall:] Incorporate PA&R process activities into program/project plans, including a program/project-unique mission assurance process map and matrix developed and maintained by the program/project with appropriate support and guidance of the Headquarters and/or Center SMA organization (Requirement). • 3.2.8.1 [SMA Managers Reporting (matrixed or direct) to the Program/Project Manager shall:] Support the program/project manager in the development and maintenance of the program Assurance Process Map and Matrix (Requirement). Governing Documents Module I There are two Documents Governing the PA&R Process NASA–wide NASA Procedural Requirement NPR 8705.6 Headquarters Office Work Instruction HOWI 8700-GE037 For complete copies of these documents, go to the NASA On-line Directive Information System (NODIS) Library at http://nodis.hq.nasa.gov/

14. The PA&R focuses on the SMA requirements lifecycle (definition, imposition, implementation, and verification). What is a PA&R? Module I The Programmatic Audit and Review (PA&R) Process is designed to provide independent verification of NASA Center, Program, or Project compliance with Agency Safety and Mission Assurance (SMA) requirements as reflected in the applicable programmatic SMA Baseline Requirement Set (BRS). The PA&R Process • Is an Agency-wide process, elements of which are owned by a variety of HQ and Center organizations. • Helps preserve and protect value created by the program or project. • Is a presence throughout the program/project lifecycle, from initial planning stages to end-of-life. • Supports Operational Decisions with Audit/Review/Assessment Reports and the identification and qualitative assessment of SMA requirements risk. A set of SMA requirements jointly negotiated among the program/project, engineering community, SMA community, and, as appropriate, institutional organizations imposed on a program/project. Typically, the BRS is a subset of Agency SMA process, technical, and engineering performance specification requirements uniquely applicable to a given NASA program, project, facility, or operation.

15. Further Elaboration on the PA&R Process Module I • What PA&R is… • An audit, assessment, or review of a program or project activity conducted by a PA&R team of Subject Matter Experts (SMEs) independent of the activity • A focused look, in both time-line and scope, to provide specific knowledge, visibility, and understanding into a program or project • PA&Rs can look at any given level of the program or project organizational hierarchy at any phase • A process that results in specific findings, corrective actions, and a final report and out-briefing to the Chief, Safety and Mission Assurance and the Office of Chief Engineer (HQ-led PA&R) or the Center SMA Director (Center-led PA&R). • What PA&R is NOT… • Continuous, day-to-day, matrixed SMA support to a program or project

16. White Hat vs. Black Hat Module I • PA&R is designed to be flexible in a “White Hat/Black Hat” application of audits, assessments and reviews. PA&R results vary by PA&R categories and include both recommendations and findings. These terms are formalized in the following slides. White Hat:Provides advice/support/mentoring to the person/organization observed. White hat assessor looks at goodness of a process. Black Hat: Assures that policies, procedures, and practices are in accordance with requirements through the implementation of rigorous audit processes.

17. PA&R Categories Module I Assessment:An evaluation and analysis of procedures, processes, and practices necessary to effectively implement requirements. The assessment findings are provided for information to the organization being assessed. Review:An examination and analysis of documentation to evaluate the feasibility, appropriateness, relevance, and/or effectiveness of documentation contents. It is a necessary precursor to an audit or assessment. The review findings are provided for information to the organization being assessed. Audit:An onsite in-process verification of compliance with policies, procedures, processes, and requirements. Audits have the responsibility for citing non-compliances, as well as observations, and require a follow-up corrective action process, including a corrective action plan and corrective action status report(s).

18. PA&R Findings Module I Critical Non-compliance (CNC): A failure to comply with a program/project SMA plan requirement, and/or a contract SMA requirement that could lead to: 1) loss of life or serious injury, and/or 2) loss of a high value asset, and/or 3) impairment of mission success. A CNC will require immediate remedial action. Non-compliance (NC):A failure to comply with a program/ project SMA plan requirement, and/or a contractually imposed SMA requirement. Observation (OBS): A finding which is technically not a non-compliance but has the potential to adversely affect safety and/or mission success. Observations may also include identification of potential best practices.

19. This section of the training generally describes how we implement the PA&R Process. It introduces the Principal Steps and how they fit into the traditional Program/Project Lifecycle How Do We Implement the PA&R Process? Module I

20. Knowledge Knowledge Develop Develop of Program SMA requirements of Program SMA requirements - - Visibility Visibility - - Gain Gain into SMA capability and compliance levels into SMA capability and compliance levels Understanding Understanding - - Communicate Communicate of associated of associated Residual Risks Residual Risks to support Operational Readiness decisions to support Operational Readiness decisions Knowledge Knowledge Visibility Visibility Understanding Understanding Support Support Make Make Conduct Conduct Understand Understand Verify Verify Verify Verify Verify Process Verify Process Operational Operational Decision Decision Program Program Baseline SMA Baseline SMA Requirements Requirements Requirements Requirements Capability Capability Readiness Readiness to Audit to Audit Discovery Discovery Requirements Requirements Flow Flow - - down down Compliance Compliance Decision Decision Provide Provide Feedback Feedback Top-Level PA&R Goals and Process Module I These are the Principal Steps of the PA&R Process. A general discussion follows. Modules II, III, and IV provide a detailed discussion of each of step.

21. Top-Level PA&R Goals and Process Module I • Make risk-based selection determined by direction, request, or formal selection logic • Finalize schedule through PA&R Planning Workgroup • Notify program or project of selection • Develop initial scope • Establish the PA&R Team

22. Top-Level PA&R Goals and Process Module I Working with Program/Project POC, compile documentation and information to learn about the program: • PA&R team briefly summarizes program or project SMA processes • PA&R reviews Project/Program developed SMA Maps and Matrices which identify SMA roles, responsibilities, and relationships

23. Top-Level PA&R Goals and Process Module I • Obtain the program-specific SMA baseline requirements set (BRS) for each program or project • Understand the risk associated with any SMA requirements gaps

24. Top-Level PA&R Goals and Process Module I • Review the formal documented flow-down of the BRS through the organizational chain (program to project to prime contractor to subcontractor) • Propose recommendations, as appropriate.

25. Top-Level PA&R Goals and Process Module I • Review key SMA Processes • Assess each implementing organization (any level in the organizational chain) to ensure there are sufficient SMA resources (i.e., staffing, skill mix, tools, funding, etc.) • Propose recommendations, as appropriate.

26. Top-Level PA&R Goals and Process Module I • Conduct a rigorous, in-process, on-site compliance verification audit to verify implementation of baseline requirements • Document audit findings which consist of non-compliances and observations • Track and verify corrective action

27. Top-Level PA&R Goals and Process Module I Inform all SMA process stakeholders of the PA&R results and any suggested revisions to Agency-wide or program-specific requirements

28. Top-Level PA&R Goals and Process Module I Summarize PA&R findings and SMA requirement risks for the Chief, Safety and Mission Assurance, the Chief Engineer or appropriate Center-level decision maker

29. PA&R Process 1. Make Decision To Audit 3. Understand Baseline SMA Requirements 4. Verify Requirements Flow-down 5. Verify Process Capability 6. Verify Requirements Compliance 7. Support Operational Readiness Decision 2. Conduct Program Discovery 7. Provide Feedback Audit / Review Opportunities Throughout Program Lifecycle Contract Award Build Review Launch SRR PDR CDR FRR LRR Program Lifecycle Mission Needs and Conceptual Studies Concept Definition Design & Development Fabrication, Integration, Test & Evaluation Pre- Operations Operations / Disposal PA&R – A Lifecycle Approach Module I • The PA&R Process is designed to be applied during any phase or before/between any major decision points of a program or project lifecycle, e.g. Program Milestone Reviews such as SRR, PDR, CRR, FRR and LRR. Emphasis typically evolves from concentration on flow-down and capability verification early in a program or project to rigorous compliance verification in later lifecycle phases. • Because no two programs/projects are identical, the PA&R Process adapts to the unique complexity of each program or project, taking full advantage of information gained in earlier phases, as well as other internal and external audits and reviews.

30. PBMA-KMS Provides Key Support to the PA&R Process Module I • The Process Based Mission Assurance Knowledge Management System (PBMA-KMS) provides quick and easy access to critical Safety and Mission Data to support program managers and SMA professionals. • PBMA-KMS functionality is being actively applied to manage the PA&R process: • Enhanced Security Work Groups are used to manage and support PA&R activities • Library functionality used to distribute team data, requirements documents, briefing materials, and audit reports to team members • Audit/Review Findings securely stored via Secure But Unclassified (SBU) capability • SecureMeeting is used to share real-time sensitive information and conduct meetings remotely via the Internet. • The Knowledge Registry is used to identify individuals with the best SMA / technical expertise to support PA&R audits/reviews. • To access PBMA go to: http://pbma.nasa.gov