1 / 21

NASA OSMA SAS '02

NASA OSMA SAS '02. Software Reliability Modeling: Traditional and Non-Parametric Dolores R. Wallace Victor Laing SRS Information Services Software Assurance Technology Center http://satc.gsfc.nasa.gov/ dwallac, vlaing@pop300.gsfc.nasa.gov. The Problem.

joelfrank
Download Presentation

NASA OSMA SAS '02

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NASA OSMA SAS '02 Software Reliability Modeling:Traditional and Non-ParametricDolores R. WallaceVictor LaingSRS Information ServicesSoftware Assurance Technology Centerhttp://satc.gsfc.nasa.gov/dwallac, vlaing@pop300.gsfc.nasa.gov NASA OSMA SAS02

  2. The Problem • Critical NASA systems must execute successfully for a specified time under specified conditions -- Reliability • Most systems rely on software • Hence, a means to measure software reliability is essential to determining readiness for operation • Software reliability modeling provides one data point for reliability measurement NASA OSMA SAS02

  3. Software Reliability Modeling(SRM) – Traditional • Captures hardware reliability engineering concepts • Mathematically models behavior of a software system from failure data to predict reliability growth • Invokes curve-fitting techniques to determine values of parameters used in the models • Validates models with data with statistical analysis • Using parametric values, predicts future measurements, e.g., • Mean time to failure • Total number faults remaining • Number faults at time t NASA OSMA SAS02

  4. Synopsis • FY01 • Identify mathematics of hardware reliability not used in software • Identify differences between hardware, software affecting reliability measurement • Identify possible improvements • FY02 • Demonstrate practicality of SRM at GSFC • Fault correction improvement – Schneidewind • Non-parametric model - Laing NASA OSMA SAS02

  5. SRM: Data Collection • Resistance to data collection • Data content • Accuracy of content • Dates of failure, correction • Calendar time not execution time • Activities/ phase when failures occur • Data manipulation • Frequency counts • Interval size and length • Time between failure NASA OSMA SAS02

  6. IntervalCounter Sample had 35 weeks – simplified fault count NASA OSMA SAS02

  7. SMERFS^3 3-D OUTPUT NASA OSMA SAS02

  8. Practical Method • SATC Services • SATC executes models and prepares analysis • SATC provides training and public domain tool • Improvements • Recommendations to projects for data collection • IntervalCounter to simplify data manipulation NASA OSMA SAS02

  9. Fault Correction Adjustments • Reliability growth occurs from fault correction • Failure correction proportional to rate of failure detection • Adjusted model with delay dT (based on queuing service) but same general form as faults detected at time T • Process: use SMERFS Schneidewind model to get parameters; apply to revised model via spreadsheet • Results • Show reliability growth due to fault correction • Predict stopping rules for testing NASA OSMA SAS02

  10. SMERFS^3 – Excel Approach* • Best approach: combine SMERFS^3 with Excel. • SRT provides model parameter estimation. • Copy and paste parameters from SRT into spreadsheet. • Excel extends capabilities of SRT by allowing user to provide equations, statistical analysis, and plots. * CASRE or other software reliability modeling tool may be used with EXCEL Recommended approach until the SRM tools incorporate this new model. NASA OSMA SAS02

  11. Non-parametric Reliability Modeling • Hardware • - Wears out over time • - Increasing failure rate • Software • - Do not wear over time • - Decreasing failure rate NASA OSMA SAS02

  12. Continued • Hardware Reliability Modeling • - “Large” independent random sampling • - Model reliability • - Make predictions • Software Reliability Modeling • - “Small” observed dependent sample (of size one?) • - Not based on independent random sampling • - Model reliability • - Make predictions? • Do we search for the silver bullet of SWR models? NASA OSMA SAS02

  13. Reliability Trending • Hardware Reliability • 100%Maximum • 0%Minimum • 0 1 2 3 4 …  • Time • Software Reliability • 100% Maximum • 0% Minimum • 0 1 2 3 4 …  • Time NASA OSMA SAS02

  14. Software Reliability Bounds • 100% Maximum • Estimated Bound • Estimated Model • 0% Minimum • 0 1 2 3 4 …  • Time NASA OSMA SAS02

  15. Calculation of Estimated Models and Bounds • Dynamic Metrics • - Failure rate data • - Problem reports • Static Code Metrics • - Traditional • - Source Lines of Code (SLOC) • - Cyclomatic Complexity (CC) • - Comment Percentage (CP) • - Object-Oriented • - Coupling Between Objects (CBO) • - Depth of Inheritance Tree (DIT) • - Weighted Methods per Class (WMC) NASA OSMA SAS02

  16. Combining Dynamic and Static Metrics • The Proportional Hazards Model (PHM) • PHM Non-Parametric Component (Static) • R(t|z) = {R0(t)}g(z) • Parametric Component (Dynamic) • - Where zβ = z1β1 + z2β2 + … + zpβp , βi’s are unknown • regression coefficients and zi’s are static code metrics data NASA OSMA SAS02

  17. Tool Schema • Input Data z = (z1, z2, … zp) • DatabaseObserved Data • Data Processing R(t|z) = {R0(t)}g(z) • Weighted Average Raw Data • Output Data Estimated Model Estimated Bound • - Process Below Bounds • Action - Corrective Action • - Process Above Bounds • - No Corrective Action NASA OSMA SAS02

  18. SUMMARY • Software reliability modeling • Provides useful measurements for decisions • Does not require expert knowledge of the math! • Is relatively easy with use of software tools • Fault correction improvement • Adapts model to be more like software • Demonstrates combined use of traditional SRM tools with spreadsheet technology • Non-parametric modeling • New approach shows promise • Prototype to be expanded NASA OSMA SAS02

  19. AIAA Recommended Steps(specific to SRM) • Characterizing the environment • Determining test approach • Selecting models • Collecting data • Estimating parameters • Validating the models • Performing analysis NASA OSMA SAS02

  20. Fault Correction Modeling • Software reliability models focus on modeling and predicting failure occurrence • There has not been equal priority on modeling the fault correction process. • Fault correction modeling and prediction support to • predict whether reliability goals have been achieved • develop stopping rules for testing • formulate test strategies • rationally allocate test resources. NASA OSMA SAS02

  21. Equations: Prediction and Comparison Worksheets Time to Next Failure(s) Predicted at Time t Remaining Failures Predicted at Time t: r(t) = (/) – Xs,t Cumulative Number of Failures Detected at Time T: D(T) = (α/β)[1 – exp (-β ((T –s + 1)))] + Xs-1 Cumulative Number of Failures Detected Over Life of Software TL: D(TL) = / + Xs-1 Equations developed by Dr. Norman Schneidewind, Naval Postgraduate School, Monterey, CA NASA OSMA SAS02

More Related