osma sas 08 infusion of perspective based inspections for nasa iv v technical briefing n.
Skip this Video
Loading SlideShow in 5 Seconds..
OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing) PowerPoint Presentation
Download Presentation
OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)

Loading in 2 Seconds...

play fullscreen
1 / 20

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing) - PowerPoint PPT Presentation

  • Uploaded on

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing). Dr. Forrest Shull, FCMD Kurt Woodham, L-3 Communications. Outline. Introduction to Perspective-Based Inspection (PBI) Tailoring of PBI for the IV&V Context Evaluation Plan. Software Inspection.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)' - mick

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
osma sas 08 infusion of perspective based inspections for nasa iv v technical briefing

OSMA SAS 08Infusion of Perspective-Based Inspections forNASA IV&V(Technical Briefing)

Dr. Forrest Shull, FCMD

Kurt Woodham, L-3 Communications

MAC-T IVV-08-162

  • Introduction to Perspective-Based Inspection (PBI)
  • Tailoring of PBI for the IV&V Context
  • Evaluation Plan

MAC-T IVV-08-162

software inspection
Software Inspection
  • A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software
    • “Cost savings rule” – Cost to find & fix software defects is about 100x more expensive after delivery than in early lifecycle phases, for certain types of defects.
      • IBM: 117:1 between code and use
      • Toshiba: 137:1between pre- and post-shipment
      • Data Analysis Center for Software: 100:1
    • “Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied.
      • 50-70% across many companies (Laitenberger)
      • 64% on large projects at Harris GCSD (Elliott)
      • 60% in PSP design/code reviews (Roy)
      • 50-95%, rising with increased discipline (O’Neill)
      • … many others

MAC-T IVV-08-162

improving inspections

Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation




Improving Inspections

































MAC-T IVV-08-162

perspective based inspection pbi
Perspective-Based Inspection (PBI)
  • A procedural technique incorporating three best practices:
    • Giving each reviewer a particular and unique focus (or perspective) on the document under review
      • Base perspectives on document stakeholders to help understand necessary participants, relate inspection tasks to development skills
    • Making individual review of a document an active (rather than passive) undertaking
    • Articulating the quality aspects of interest

MAC-T IVV-08-162

perspective based inspection pbi1
Perspective-Based Inspection (PBI)
  • History
    • 2001-2003: SARP research funding
      • Refined basic approach, implemented on NASA projects and in classrooms, measured results
    • 2004: Tech infusion with Flight Software Branch / GSFC
      • Goal: Produce new Branch standards
      • Success metric: Based on experiences on pilot(s), how much of the recommended process is adopted by Branch?
    • 2004: Tech infusion with United Space Alliance / JSC
      • Goal: Reduce defect slippage over current increment
      • Success metric: Show reduced defect slippage on PB-inspected artifacts
    • 2007-2009: SARP funding for improved general inspection planning
  • Other Results and Benefits
    • Improved defect detection effectiveness substantially (NASA & elsewhere)
    • Helped re-invigorate inspection practices
    • Forms the basis of industrial and academic training

MAC-T IVV-08-162

adopting pbi for iv v
Adopting PBI for IV&V
  • Pathfinder: Infuse PBI into the IV&V practices being applied to JWST
    • JWST Project/Developer artifact review
    • JWST IV&V System Reference Model (SRM) validation
  • Technology Provider…
    • Adapts PBI to IV&V domain and JWST-specific context
    • Administers training
    • Facilitates initial reviews
    • Monitors efficacy of PBI and documents results/recommendations
  • JWST IV&V Team…
    • Receives training
    • Conducts reviews in-line with IV&V execution
    • Provides ancillary feedback to Technology Provider (minimal disruption)
  • IV&V Facility Research…
    • Monitors progress and determines next-step expansion based on JWST pathfinder

MAC-T IVV-08-162

jwst system overview
JWST System Overview
  • Launch in 2013
  • Collaboration between NASA, ESA, CSA
  • Managed by GSFC
  • NGST – prime contractor
  • STScI – operator after launch
  • Integrated Instruments
  • Mid-Infrared Instrument (MIRI)
  • Near-Infrared Spectrograph (NIRSpec)
  • Near-Infrared Camera (NIRCam)
  • Fine Guidance Sensor (FGS)

MAC-T IVV-08-162

jwst observatory diagram
JWST Observatory Diagram

Image credit: http://www.jwst.nasa.gov/observatory.html

MAC-T IVV-08-162

jwst iv v overview
JWST IV&V Overview
  • IV&V Scope
    • Observatory and Ground System segments
  • SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels
    • Specific focus areas driven by IV&V criticality/risk analysis with close coordination with JWST project
  • WBS driven
      • SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html)
      • Defines validation and verification tasking to be performed at various levels of system decomposition
        • Emphasis on early validation of requirements with respect to system needs, goals, and objectives in light of three validation questions
        • Validated SRM plays pivotal role in validation and verification activities throughout IV&V life-cycle

MAC-T IVV-08-162

adopting pbi for iv v perspectives

Document with defects

High overlap Low coverage

(Ad hoc)

Low overlap

High coverage


Adopting PBI for IV&V: Perspectives
  • Motivation:
    • Focus the responsibilities of each inspector
    • Minimize overlap among inspector responsibilities
    • Maximize union of defects found

MAC-T IVV-08-162

adopting pbi for iv v perspectives1

System Ref Model

Adopting PBI for IV&V: Perspectives
  • Tailoring required for new domain, new issues, new artifacts

Is model complete, correct, consistent, and testable?

Model validation

Model construction

Needs, goals, objectives elucidation

Analysis val./ver. target

Do sources adequately define user needs?

Do behaviors defined in val./ver. targets map to equivalent definition in model?

MAC-T IVV-08-162


Adopting PBI for IV&V: Defect Types

  • Interpretation of “defect”
    • Avoid focusing on correctness as the one and only quality property
    • Any issue that would cause a problem for a stakeholder downstream.

MAC-T IVV-08-162

adopting pbi for iv v defect types
Adopting PBI for IV&V: Defect Types
  • Other sources of defect categories:
    • Checklists and defect categories from previous work
    • Experienced inspectors
      • What is it that the most experienced people are looking for?
      • Can get from:
        • Asking them!
        • Noting recurring questions at inspection meetings
    • Defect/discrepancy report databases
      • What types of problems seem to be common?
      • What types of problems take the longest to close?

MAC-T IVV-08-162

adopting pbi for iv v active review
Adopting PBI for IV&V: Active Review
  • Reading through a document one page at a time is not the most effective approach.
  • In PBI every inspector receives a scenario to guide his/her work.
    • Based on your perspective…
    • Imagine you are creating some other artifact you need as part of your job…
      • E.g. for tester perspective, think about building a test plan
    • Then read through the document as you normally would to get the necessary information.
  • Inspectors have questions about quality to answer while they follow the scenario.

MAC-T IVV-08-162

adopting pbi for iv v summary
Adopting PBI for IV&V: Summary
  • Initial perspectives for this domain:
    • Modeler
      • Uses requirements & existing models to create the next iteration
      • Quality foci: Appropriate level of detail in requirements; completeness and clarity of requirements & models; consistency and correctness of models
    • Domain expert
      • Assesses whether model accurately and usefully captures domain
      • Quality foci: Correctness of requirements (independent of models); identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components
    • Quality assurance
      • Assesses whether models and system can be validated properly
      • Quality foci: Handling of exception cases / unexpected system conditions; robustness of system; testability

MAC-T IVV-08-162

infusion plan
Infusion Plan
  • Understand team-specific quality concerns
  • Understand likely perspectives for the team
  • Refine set of perspectives and define procedures for each. E.g.,
    • What quality concerns map to which perspectives
    • What is a feasible/effective process for checking those concerns
  • Finalize procedures and provide training
  • Analyze ongoing inspection results (and update procedures if necessary)
  • Monitor downstream defect profile
  • Write final report

Status: Reviewed sample models & processes; Constructed draft PBI techniques; Created draft training course

MAC-T IVV-08-162

infusion plan path forward
Infusion Plan: Path Forward
  • Evaluation
    • Success metric: Effectiveness of PB inspection in comparison to past inspections
  • Current JWST approach
    • Project documents
      • Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts
    • SRM Reviews
      • IV&V team maintains SRM change logs, action item lists, and other sources for the infusion team to evaluate relative effectiveness
      • IV&V team documents discrepancy when comparing SRM against val/ver target
  • Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items
    • Likely to involve follow-up conversation/interviews to assess results and collect ancillary observations from the IV&V team
    • Will balance this with objective to minimize impact to IV&V activities

MAC-T IVV-08-162

future considerations
Future Considerations
  • Use measurement results to further optimize, e.g.,
    • Consistently missing a certain type of defect?
      • Add a new defect type and associated questions to relevant perspectives
    • Missing a whole set of defects?
      • Consider whether a new perspective should be added.
    • Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find?
      • Consider deleting the relevant defect type or perspective.
    • Do we consistently face a lot of problems of type x?
      • Add more reviewers using relevant perspectives

MAC-T IVV-08-162

contact information
Contact information

Forrest Shull



Kurt Woodham



MAC-T IVV-08-162