1 / 19

SAS 2007 Executive Galaxy Global Corporation Presented by: Valerie Jones & Jennifer Murray

Evaluation of Current Requirements Analysis Tools Capabilities for IV&V in the Requirements Analysis Phase. SAS 2007 Executive Galaxy Global Corporation Presented by: Valerie Jones & Jennifer Murray NASA POC: Jeffrey Northey SAS_07_EvalRaTools_RAPhase_Jones2. Problem Statement.

gchristie
Download Presentation

SAS 2007 Executive Galaxy Global Corporation Presented by: Valerie Jones & Jennifer Murray

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Evaluation of Current Requirements Analysis Tools Capabilities for IV&V in the Requirements Analysis Phase SAS 2007 Executive Galaxy Global Corporation Presented by: Valerie Jones & Jennifer Murray NASA POC: Jeffrey Northey SAS_07_EvalRaTools_RAPhase_Jones2

  2. Problem Statement • A Requirement Analysis tool that performs comprehensive automated analysis does not exist • Existing tools available for use do not provide guidance or automated assistance in identifying potentially problematic requirements • Requirements analysis is a long and tedious process. • An automated tool could decrease the time required during the requirements analysis phase as well as improve the process.

  3. Tool Evaluation Study • Tools Evaluated: • NASA E-Smart/ARM • Lexior • QuARS • Requirements Assistant • SAT • TEKChecker • Tools evaluated on a predefined set of technical and quality attributes. • Model problem defined to do comparison between manual analysis and analysis with automated assistance

  4. Tool Study Results • Two evaluation studies were performed • Results indicated automated analysis tools correctly identified technical issues consistent with manual analysis. • Requirements Assistant performed best technically in both studies • Analysis results were consistent with manual analysis and additional key issues identified • Key issues identified including: missing requirements and inconsistencies in requirements

  5. RA Technical Results • Comparing to original 21 TIMs submitted for project. • 12/21 were found correctly. • 6/21 were flagged but for other reasons. • 5/21 were not flagged at all. • In addition to matching TIM’s generated, tool also found missed defects that were not reported originally. • RA was the only tool to identify missing and inconsistent requirements

  6. Technical Results (Project 2) • Requirements Assistant (RA) • Correctly Identified 2 RIDs. • Could possible collect additional 2 RIDs if entire document was analyzed. • One was found via traceability. (Outside of tools capabilities) • Additional 13 issues identified and presented to project manager

  7. RA Differences • RA was the only tool evaluated that takes domain information specific to the system into consideration • Allows for creation of a domain corpus to define known information about the system • Input files allow tool to identify missing or inconsistent requirements • RA analyzes starting at the sentence level, paragraph level then entire document as a whole. • Utilizes knowledgebase of rules developed of 25 years of experience on various systems

  8. File Definitions • Milfile - This file contains the aspects that the reviewer wants to address. • Milsub - an entry, that should be considered when the word “system” is found in a requirement. • Actor - an actor in the system (not necessarily humans). • Actuator - a list of hardware that the actor(s) work on. • Quant – Units of measure • Function - An array of functions that the system has, or should have (in the reviewer’s opinion). • Gloss - This contains the glossary of terms in the requirements document. (if provided). The file only contains the glossary terms, not their definition. • List_Sensor – A list of sensors known in the system.

  9. Tool Pre-Processing • Input document must be in standard text format • Input files are currently entered manually • Input files are not required but increase the tools ability to correctly identify errors

  10. Tool Output • Examples – Poorly written requirements1. Appropriate system standards shall be used where necessary.Remarks of The Requirements Assistant™:a. appropriate – What is appropriate?b. standards – Specifically which standards are required?c. where necessary – Under what conditions will these standards be applied?2. It is required that the red light be illuminated once temperatures exceed about 25 degrees. Remarks of The Requirements Assistant™:a. It is required – Is this a requirement?b. Illuminated – Is this for a certain duration?c. Temperatures – temperatures of..?d. About – accuracy?e. Degrees – Celsius, Fahrenheit?f. Is there also a requirement how/when the red light will be switched off?

  11. Examples cont. • Off-line tooling allowing the end-user to manipulate … in order to… compensate for … stability.Remarks of The Requirements Assistant™:a. manipulate – exactly what is manipulated?b. In order to – this is not a requirement but a goal.c. Compensate for – how much? • Remarks of The Requirements Assistant™:No requirements on reliability, safety and EMC were found in the set of requirements.

  12. Various Output Previews • The Requirements Assistant™ provides three different previews of its analysis results: View by quality attributes: a. Consistency b. Completeness c. Testability d. Accuracy e. Readability

  13. Various Output Options cont. View by sentence: • Poor words (in bold): The FSW will send appropriate error messages, and set the fault condition to “failed”, should the alive and Idle discrete lines fail to signal ready ” and “Idle” within a parameterized amount of time [General Remark: 4 Upper limit? General Remark: and, and, and, complex sentence ] • The FSW will send commands, and receive health & status and science telemetry information from most XXX instrumentation through this interface.General Remark: Quantity-like in paragraph most in sentenceNon Requirement ["will"]: null • View by topic • DELIVERY AND RETURN "The XXX shall deliver at least 2850 kg (6,283 lbm) (gross) of cargo. • DELIVERY AND RETURN "The XXX shall return at least 2,858 kg (6,283 lbm) of cargo.

  14. Various Output Options cont. • View by concept: Cargo Processing Concept: • DELIVERY AND RETURN "The XXX shall deliver at least 2850 kg (6,283 lbm) (gross) of cargo. • DELIVERY AND RETURN "The XXX shall return at least 2,858 kg (6,283 lbm) of cargo.

  15. Benefits of RA • Leads analyst to identify key issues outside capabilities of other tools evaluated • Could allow the analyst to prioritize analysis based on identification of potentially problematic requirements first. • Better results for both novice and experienced analysts • Novice analyst identified key issues with a total analysis time of 3 days compared to a team of analysts working for 3 weeks a (50% decrease in analysis time). • Developers would have positive ROI if using this tool for tracking requirements across stages of development • Can assist in Requirements Validation with UML modeling. • Knowledge base behind the tool allows for very versatile use for various aspects of software development to stay consistent

  16. Recommendations • Proposed facility adopt Requirements Assistant (RA) for further use. • RA requires additional upgrades before it can be used facility wide. • Upgrades • Improve user interface for analyst efficiency • Increase speed • UML support • Domain term library

  17. Proof of Concept • Current tasking is to evaluate RA capabilities for requirements validation and verification to UML modeling of systems. • Working with NASA IV&V Facility project performing requirements validation and verification of 4 System Reference Models utilizing RA capabilities. • Initial results from first model analyzed were provided to the project • Project implemented additional updates based on findings • Through the POC we will work to develop an appropriate user interface for the tool and the IV&V analyst.

  18. Proof of Concept Model

  19. What’s Next? • Continue the Proof of Concept Study for Requirements Validation • Recommended updates to Requirements Assistant for user interface development. • Evaluating Eclipse platform • Develop additional requirements. • Proposal submitted for Research Infusion.

More Related