Michael Kass
Download
1 / 10

Software Assurance Metrics and Tool Evaluation (SAMATE) - PowerPoint PPT Presentation


  • 65 Views
  • Uploaded on

Michael Kass National Institute of Standards and Technology http://samate.nist.gov/ [email protected] Software Assurance Metrics and Tool Evaluation (SAMATE). Outline . Overview of Software Assurance (SwA) tool testing at NIST Description of SAMATE project Follow-on.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Software Assurance Metrics and Tool Evaluation (SAMATE)' - ramona-may


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Michael Kass

National Institute of Standards and Technology

http://samate.nist.gov/

[email protected]

Software Assurance Metrics and Tool Evaluation (SAMATE)


Outline
Outline

Overview of Software Assurance (SwA) tool testing at NIST

Description of SAMATE project

Follow-on


Dept homeland security concern
Dept Homeland Security Concern

Do software assurance tools work as they should?

Do they really find vulnerabilities and catch bugs? How much assurance does running the tool provide?

Software Assurance tools should be:

Tested (accurate and reliable)

Peer reviewed

Generally accepted


Goals of samate
Goals of SAMATE

Develop metrics for the effectiveness of SwA tools and to identify deficiencies in software assurance methods and tools

Perform SwA R&D to assess current methods and tools in order to identify deficiencies which can lead to software product failures and vulnerabilities

Identify gaps in methods and tools and suggest areas of research


The nist samate project software assurance metrics and tool evaluation
The NIST SAMATE Project(Software Assurance Metrics and Tool Evaluation)

Conduct surveys

Tools

Researchers and companies

Host workshops & conference sessions

Taxonomy of SwA functions and techniques

Order of importance (cost/benefit, criticalities, …)

Gaps and research agendas

Studies to develop tool effectiveness metrics

Evaluate tools

Detailed specification

Test plans

Host reference dataset library


A taxonomy of static analysis tool functions
A Taxonomy of Static Analysis Tool Functions

  • Language

  • Source/Binary analysis

  • Semantic checking (abstract syntax tree)

  • Interprocedural analysis

  • Strong type checking (type casting vulnerabilities, uninitialized variable use)

  • Memory allocation checking (memory leaks, deallocation of unallocated memory)

  • Logic checking (unnecessary code, unreachable code)

  • Interface checking (include file cycling)

  • Security checking

    • Buffer overflow/underflow

      • Stack overflows

      • Heap overflows

    • Integer overflow/underflow

    • Tainted data

    • Error path problems

    • Locking problems

  • Code metric generation (LOC, number of methods, levels of inheritance)


Sa tool effectiveness metrics
SA Tool Effectiveness Metrics

What constitutes a tool’s effectiveness metric?

  • Number of defects detected vs. total defects

  • Number of false positives

  • Number of false negatives


Documenting tool effectiveness
Documenting tool effectiveness

  • Tool functional specification

  • Test plan

  • Reference dataset

  • Test report


Workshop1

SA

classes

Workshop 2

fill

gaps

Workshop 3

Define

Metric

focus

group

class 1

focus

group

class 2

focus

group

class 2

focus

group

class 1

SAMATE Project Timeline

T(mos.)

1

2

3

4

5

6

9

12

15

18

21

24

Tool

Survey

Function

Taxonomy

Survey

Publication

tool

testing

matrix

Spec0

test reports

select

func

test plan

Spec1

strawman

spec

test plan

draft

test reports

select func

Spec0

test reports

test plan

Spec1

strawman

spec

test plan

test reports

draft


Contact for samate participation
Contact for SAMATE Participation

Paul Black

Project Leader, Software Diagnostics & Conformance Testing Division, Software Quality Group

[email protected]


ad