software assurance metrics and tool evaluation l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Software Assurance Metrics and Tool Evaluation PowerPoint Presentation
Download Presentation
Software Assurance Metrics and Tool Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 20

Software Assurance Metrics and Tool Evaluation - PowerPoint PPT Presentation


  • 124 Views
  • Uploaded on

Software Assurance Metrics and Tool Evaluation. Paul E. Black National Institute of Standards and Technology http://www.nist.gov/ paul.black@nist.gov. Outline. Introduction SRD project SAMATE project structure Future – where do we go now?. Good Software. Good Checking.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Software Assurance Metrics and Tool Evaluation' - freja


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
software assurance metrics and tool evaluation

Software Assurance Metrics and Tool Evaluation

Paul E. Black

National Institute of Standards and Technology

http://www.nist.gov/

paul.black@nist.gov

outline
Outline
  • Introduction
  • SRD project
  • SAMATE project structure
  • Future – where do we go now?

Paul E. Black

what is software assurance

Good

Software

Good Checking

Good Development

What is Software Assurance?
  • Activities that ensures that software processes and products conform to requirements.
        • after NASA Software Assurance Guidebook
  • Two legs of good software
    • Good Development
    • Good Checking
      • Testing (dynamic)
      • Analysis (static)

Paul E. Black

nist s role
NIST’s role
  • What is NIST?
    • A non-regulatory agency in Dept. of Commerce
    • 3,000 employees in Maryland and Colorado
    • Primarily research, not funding
  • Why NIST?
    • Over a century of experience in standards and measurement
    • Involved in security: DES, AES, NVLAP, etc.
    • Trusted, neutral 3rd party

Paul E. Black

the two projects
The Two Projects

SAMATE

Software Assurance Metrics

And Tool Evaluation

SRD

Standard Reference

Dataset

Paul E. Black

outline6
Outline
  • Introduction
  • SRD project
  • SAMATE project structure
  • Future – where do we go now?

Paul E. Black

srd project goals
SRD Project Goals
  • Identify classes of security flaws and vulnerabilities
  • Identify classes of software security assessment techniques
  • Document state of the art
  • Develop a Standard Reference Dataset (SRD) of clean programs and programs with security flaws

Paul E. Black

srd characteristics
SRD Characteristics
  • Small test cases for each flaw

Separate “can detect” from speed issues

  • Flawed programs and their clean counterparts
  • Very large test cases

Confirm speed and maximum size

  • Test cases taken from actual code

Nobody can say it would never happen

  • Many different subsets

Java, C, web app, OS, Windows, Unix, etc.

  • Ongoing development and additions
  • Submissions from NIST, researchers, & vendors
  • Readily usable

Paul E. Black

srd project plans
SRD Project Plans
  • Small workshop

http://samate.nist.gov/softSecToolsSOA

    • 10 & 11 August at NIST
    • Publish proceedings as NIST Special Publication & put in ACM Digital Library
  • Write journal article on
    • classes of known software security vulnerabilities and
    • the state of the art of security SA tools

Paul E. Black

outline10
Outline
  • Introduction
  • SRD project
  • SAMATE project structure
  • Future – where do we go now?

Paul E. Black

dhs software assurance plan
DHS Software Assurance Plan

1. PEOPLE (Education/Training)

Software Developer focused training and education

2. PROCESS (Lifecycle, Best Practices, Standards)

Security throughout the Software Development Life Cycle

3. TECHNOLOGY (Tools and R&D)

SA tools identification, enhancement, and development

4. ACQUISITION (SOW / Procurement language)

Embed security requirements in procurement stage

Paul E. Black

the samate project
The SAMATE Project

http://samate.nist.gov/

  • Compendiums (ongoing)
    • Tools
    • Researchers and companies
  • Workshops
  • Aids for tool evaluation
  • Software metrics

Paul E. Black

workshops
Workshops
  • Taxonomy of SA functions and techniques
    • Approach (code vs. spec, static vs. dynamic)
    • Software type (distributed, real time, secure)
    • Type of fault detected
  • Which are the most “important”?
    • Highest cost/benefit ratio?
    • Finds highest priority vulnerabilities?
  • Identify gaps in SA functions and write research agenda
  • Plan and initiate studies for metrics

first workshop in Long Beach, Nov, w/ASE

Paul E. Black

purposes of sa tool evaluations
Purposes of SA Tool Evaluations
  • Precisely document what a tool does (and doesn’t) do

… in order to …

  • Provide feedback to tool developers
    • Simple changes to make
    • Directions for future releases
  • Inform users
    • Match the tool to a particular situation
    • Understand significance of tool results
  • Guide research for next tool generation

Paul E. Black

developing a specification
Developing a Specification
  • After tool function selection approved by working group, …
  • NIST develops a specification for the function with focus group input
  • Spec posted to web for public comment
  • NIST develops the tests
    • Detailed plans
    • Scripts
    • Standard Reference Dataset

Paul E. Black

outline16
Outline
  • Introduction
  • SRD project
  • SAMATE project structure
  • Future – where do we go now?

Paul E. Black

toward software metrics
Toward Software Metrics
  • Qualitative comparison
  • Formally defined quantity
  • Unit and scale
  • Measured value
  • Derived units

Paul E. Black

tool effectiveness metrics
Tool Effectiveness Metrics
  • Do they really find vulnerabilities and catch bugs?
  • In other words, how much assurance does running a tool provide?
  • “Create studies and experiments to measure the effectiveness of tools”

Paul E. Black

call for participation
Call for Participation
  • Define classes of flaws and vulnerabilities
  • Contribute to collections of tools, researchers, …
  • Help define classes of SA functions
  • Decide order of importance of functions
  • Participate in focus group to specify a function
  • Contribute to standard reference dataset
  • Develop metrics to assess software and tools
  • Help set research agenda

Paul E. Black

society has 3 options
Society has 3 options:
  • Learn how to make software that works
  • Limit size or authority of software
  • Accept failing software

Paul E. Black