1 / 9

The NIST SAMATE and Evaluating Static Analysis Tools

The NIST SAMATE and Evaluating Static Analysis Tools. Paul E. Black National Institute of Standards and Technology http://www.nist.gov/ paul.black@nist.gov. What is NIST?. U.S. National Institute of Standards and Technology A non-regulatory agency in Dept. of Commerce

sgrimmett
Download Presentation

The NIST SAMATE and Evaluating Static Analysis Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The NIST SAMATE and Evaluating Static Analysis Tools Paul E. Black National Institute of Standards and Technology http://www.nist.gov/ paul.black@nist.gov

  2. What is NIST? • U.S. National Institute of Standards and Technology • A non-regulatory agency in Dept. of Commerce • 3,000 employees + adjuncts • Gaithersburg, Maryland and Boulder, Colorado • Primarily research, not funding • Over 100 years experience in standards and measurements: from dental ceramics to microspheres, from quantum computers to fire codes, from body armor to DNA forensics Paul E. Black

  3. The NIST SAMATE Project • Sponsored in part by DHS beginning July 2004 to improve Software Assurance (SwA), i.e., to • assess SwA methods and tools to identify what can prevent failures and vulnerabilities • develop metrics for their effectiveness • identify gaps in SwA methods and tools • Current areas of concentration • Source code security analyzers • Web application scanners • Binary analyzers • Software labels • Web site http://samate.nist.gov/ Paul E. Black

  4. Why Source Code Analysis? • Too few higher level tools of one kind to standardization • Half of security weaknesses are introduced during coding • Vital when software development is not visible, e.g., legacy or contract software • Feedback for process improvement • Why static analysis? Testing cannot find special cases, e.g., user name “matahari” gives full access Paul E. Black

  5. Doing Something Useful • How can we contribute? • Metric for static analysis? • Detail what should be done (“high bar”) • Set minimum and clearly say tools do much better • Vendors • Create a win/win situation • Work to earn their trust • Users, academicians, government, others • Workshops & conference sessions • E-mail discussion list • Lots of personal contact Paul E. Black

  6. Doing the Work • We (NIST) does the research and writing • For specification, test plan, and test material • Invite review & feedback piece by piece • Publicize and discuss • Visit them and adapt to their needs • Take every comment seriously Paul E. Black

  7. # vulnerabilities time fixing weaknesses reported by tools Do static analysis tools really help? # vulnerabilities time fixing weaknesses reported by tools Paul E. Black

  8. DHS funds Fortify to scan open source. Results shared with developers. Some fix weaknesses found. We use reported vulnerabilities as a surrogate for security. Compare number of vulnerabilities before and after scans. Study Dawson Engler’s Question MySQL vulnerabilities before and after version 4.1.10 Not enough data yet to support or refute hypothesis Paul E. Black

  9. Seeking Participants • Add test cases (in Ada :-) to SRD • Technical Advisory Panel • Long term direction of the SAMATE project • Focus groups to comment on specs & tests for • Source code security analyzers • Web application scanners • Static binary analyzers • contact Paul E. Black SAMATE Project Leader paul.black@nist.gov Paul E. Black

More Related