1 / 20

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing). Dr. Forrest Shull, FCMD Kurt Woodham, L-3 Communications. Outline. Introduction to Perspective-Based Inspection (PBI) Tailoring of PBI for the IV&V Context Evaluation Plan. Software Inspection.

Download Presentation

OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. OSMA SAS 08Infusion of Perspective-Based Inspections forNASA IV&V(Technical Briefing) Dr. Forrest Shull, FCMD Kurt Woodham, L-3 Communications MAC-T IVV-08-162

  2. Outline • Introduction to Perspective-Based Inspection (PBI) • Tailoring of PBI for the IV&V Context • Evaluation Plan MAC-T IVV-08-162

  3. Software Inspection • A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software • “Cost savings rule” – Cost to find & fix software defects is about 100x more expensive after delivery than in early lifecycle phases, for certain types of defects. • IBM: 117:1 between code and use • Toshiba: 137:1between pre- and post-shipment • Data Analysis Center for Software: 100:1 • “Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied. • 50-70% across many companies (Laitenberger) • 64% on large projects at Harris GCSD (Elliott) • 60% in PSP design/code reviews (Roy) • 50-95%, rising with increased discipline (O’Neill) • … many others MAC-T IVV-08-162

  4. Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation Roles Activities Products Improving Inspections Software Artifact 1 PlanningForm Planning organizer Defect Report Form 2 Preparation inspector 3 Defect Collection Form moderator inspectors author Meeting 4 Corrected Software Artifact Defect Correction Form Follow- through Software Inspection author MAC-T IVV-08-162

  5. Perspective-Based Inspection (PBI) • A procedural technique incorporating three best practices: • Giving each reviewer a particular and unique focus (or perspective) on the document under review • Base perspectives on document stakeholders to help understand necessary participants, relate inspection tasks to development skills • Making individual review of a document an active (rather than passive) undertaking • Articulating the quality aspects of interest MAC-T IVV-08-162

  6. Perspective-Based Inspection (PBI) • History • 2001-2003: SARP research funding • Refined basic approach, implemented on NASA projects and in classrooms, measured results • 2004: Tech infusion with Flight Software Branch / GSFC • Goal: Produce new Branch standards • Success metric: Based on experiences on pilot(s), how much of the recommended process is adopted by Branch? • 2004: Tech infusion with United Space Alliance / JSC • Goal: Reduce defect slippage over current increment • Success metric: Show reduced defect slippage on PB-inspected artifacts • 2007-2009: SARP funding for improved general inspection planning • Other Results and Benefits • Improved defect detection effectiveness substantially (NASA & elsewhere) • Helped re-invigorate inspection practices • Forms the basis of industrial and academic training MAC-T IVV-08-162

  7. Adopting PBI for IV&V • Pathfinder: Infuse PBI into the IV&V practices being applied to JWST • JWST Project/Developer artifact review • JWST IV&V System Reference Model (SRM) validation • Technology Provider… • Adapts PBI to IV&V domain and JWST-specific context • Administers training • Facilitates initial reviews • Monitors efficacy of PBI and documents results/recommendations • JWST IV&V Team… • Receives training • Conducts reviews in-line with IV&V execution • Provides ancillary feedback to Technology Provider (minimal disruption) • IV&V Facility Research… • Monitors progress and determines next-step expansion based on JWST pathfinder MAC-T IVV-08-162

  8. JWST System Overview • Launch in 2013 • Collaboration between NASA, ESA, CSA • Managed by GSFC • NGST – prime contractor • STScI – operator after launch • Integrated Instruments • Mid-Infrared Instrument (MIRI) • Near-Infrared Spectrograph (NIRSpec) • Near-Infrared Camera (NIRCam) • Fine Guidance Sensor (FGS) MAC-T IVV-08-162

  9. JWST Observatory Diagram Image credit: http://www.jwst.nasa.gov/observatory.html MAC-T IVV-08-162

  10. JWST IV&V Overview • IV&V Scope • Observatory and Ground System segments • SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels • Specific focus areas driven by IV&V criticality/risk analysis with close coordination with JWST project • WBS driven • SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html) • Defines validation and verification tasking to be performed at various levels of system decomposition • Emphasis on early validation of requirements with respect to system needs, goals, and objectives in light of three validation questions • Validated SRM plays pivotal role in validation and verification activities throughout IV&V life-cycle MAC-T IVV-08-162

  11. Document with defects High overlap Low coverage (Ad hoc) Low overlap High coverage (PBI) Adopting PBI for IV&V: Perspectives • Motivation: • Focus the responsibilities of each inspector • Minimize overlap among inspector responsibilities • Maximize union of defects found MAC-T IVV-08-162

  12. System Ref Model Adopting PBI for IV&V: Perspectives • Tailoring required for new domain, new issues, new artifacts Is model complete, correct, consistent, and testable? Model validation Model construction Needs, goals, objectives elucidation Analysis val./ver. target Do sources adequately define user needs? Do behaviors defined in val./ver. targets map to equivalent definition in model? MAC-T IVV-08-162

  13. Adopting PBI for IV&V: Defect Types • Interpretation of “defect” • Avoid focusing on correctness as the one and only quality property • Any issue that would cause a problem for a stakeholder downstream. MAC-T IVV-08-162

  14. Adopting PBI for IV&V: Defect Types • Other sources of defect categories: • Checklists and defect categories from previous work • Experienced inspectors • What is it that the most experienced people are looking for? • Can get from: • Asking them! • Noting recurring questions at inspection meetings • Defect/discrepancy report databases • What types of problems seem to be common? • What types of problems take the longest to close? MAC-T IVV-08-162

  15. Adopting PBI for IV&V: Active Review • Reading through a document one page at a time is not the most effective approach. • In PBI every inspector receives a scenario to guide his/her work. • Based on your perspective… • Imagine you are creating some other artifact you need as part of your job… • E.g. for tester perspective, think about building a test plan • Then read through the document as you normally would to get the necessary information. • Inspectors have questions about quality to answer while they follow the scenario. MAC-T IVV-08-162

  16. Adopting PBI for IV&V: Summary • Initial perspectives for this domain: • Modeler • Uses requirements & existing models to create the next iteration • Quality foci: Appropriate level of detail in requirements; completeness and clarity of requirements & models; consistency and correctness of models • Domain expert • Assesses whether model accurately and usefully captures domain • Quality foci: Correctness of requirements (independent of models); identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components • Quality assurance • Assesses whether models and system can be validated properly • Quality foci: Handling of exception cases / unexpected system conditions; robustness of system; testability MAC-T IVV-08-162

  17. Infusion Plan • Understand team-specific quality concerns • Understand likely perspectives for the team • Refine set of perspectives and define procedures for each. E.g., • What quality concerns map to which perspectives • What is a feasible/effective process for checking those concerns • Finalize procedures and provide training • Analyze ongoing inspection results (and update procedures if necessary) • Monitor downstream defect profile • Write final report Status: Reviewed sample models & processes; Constructed draft PBI techniques; Created draft training course MAC-T IVV-08-162

  18. Infusion Plan: Path Forward • Evaluation • Success metric: Effectiveness of PB inspection in comparison to past inspections • Current JWST approach • Project documents • Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts • SRM Reviews • IV&V team maintains SRM change logs, action item lists, and other sources for the infusion team to evaluate relative effectiveness • IV&V team documents discrepancy when comparing SRM against val/ver target • Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items • Likely to involve follow-up conversation/interviews to assess results and collect ancillary observations from the IV&V team • Will balance this with objective to minimize impact to IV&V activities MAC-T IVV-08-162

  19. Future Considerations • Use measurement results to further optimize, e.g., • Consistently missing a certain type of defect? • Add a new defect type and associated questions to relevant perspectives • Missing a whole set of defects? • Consider whether a new perspective should be added. • Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find? • Consider deleting the relevant defect type or perspective. • Do we consistently face a lot of problems of type x? • Add more reviewers using relevant perspectives MAC-T IVV-08-162

  20. Contact information Forrest Shull fshull@fc-md.umd.edu 301-403-8970 Kurt Woodham Kurt.Woodham@L-3com.com 757-644-5807 MAC-T IVV-08-162

More Related