Estimating the Cost and Benefits of Software Assurance Investments

1. Estimating the Cost and Benefits of Software Assurance Investments Thomas P. Frazier November 9, 2006

2. Background • Enemies of our country may try subverting government systems by intentionally embedding malicious code or by remotely exploiting unintentionally vulnerable software. • This risk is rising with the broadened use of commercial off-the-shelf products and the increasing globalization of software development. • Estimate the costs and benefits associated with specific policies and procedures to mitigate these risks.

3. Business Case • Business case requires that we turn the benefits into dollar values. • A software assurance investment should be undertaken if the cost of this protection is less than the expected loss of not protecting • This is done by computing the expected value of costs and benefits • The expected value is defined as E = p*L, where p is the probability of a possible outcome and L is the payoff or dollar value of such an outcome.

4. Cost-Benefit Criterion • An investment in assurance should be undertaken if C < p*L where: C =the cost associated with the software assurance investment p = probability of a successful attack by an intruder L =dollar loss resulting from malevolent intrusion Software assurance investment should be undertaken if the [present value of the] cost of this protection is less than the [present value of the] expected loss of not protecting

5. Concerns • Many of the policies and procedures are likely to be so general that merely enumerating all the cost and benefit elements is hard, much less assigning values to them. • Policies, procedures, and investment ideas will be correlated, meaning that implementing A could affect the costs and benefits associated with B. • Quantifying C, and especially p and L

6. Literature Search is Not Encouraging • Cashell, Brian, William D. Jackson, Mark Jickling, and Baird Webel, The Economic Impact of Cyber-Attacks, Congressional Research Service, Library of Congress, April 1, 2004 “No one in the field is satisfied with our present ability to measure the costs and probabilities of cyber attacks. There are no standard methodologies for cost measurement, and study of the frequency of attacks is hindered by the reluctance of organizations to make public their experiences with security breaches.” • Howard E. Glavin, “A Risk Modeling Methodology,” Computer Security Journal, Vol. 19, No. 3 (Summer 2003). • U.S. House of Representatives, Select Committee on Homeland Security, Subcommittee on Cybersecurity, Science, and Research & Development, “Cybersecurity for the Homeland,” December 2004.

7. Proposed Approach • Use some combination of data taken from the U.S. liability insurance industry, combined with estimates from experts to estimate p and L • Underwriters essentially estimate p and L every time they write a policy. • Employ liability insurance data from firms that write and implement software

8. Liability Insurance Data • Data from the product liability industry can shed some light on the following: • How much liability insurance is purchased by firms of various industries and sizes, • How much the firms pay for the insurance, and • How much the firms set aside for self-insurance, etc. (i.e., exclusions and other coverage information)

9. An Example • Assume that one of the recommended investments is a “red team” review of key software. • C - cost of performing the red team reviews, measured by the fully burdened cost of the red team members plus some estimate of the other direct charges typically incurred in these types of operations (e.g., 15 percent of the direct labor charges). COSECMO model may also prove to be useful in estimating C.

10. Estimating Risk • Assume that the cost of insurance (C) is the expected loss coverage (L) times the probability of loss (p) • That is, C = pL or p = C÷L • p may overstate the probability since the insurance company makes a profit • However, the policyholder retains some risk (deductible) which means the actual loss is usually greater than the expected insured loss in the above relationship • For simplicity, we will assume these effects offset each other • Insurance data on premiums paid for typical coverage indicate that software companies pay about $5 million in annual premiums (C) for $1 billion of coverage (L). • p = C÷L = $5M ÷ $1B = 0.005 • This leads us to estimate that the probability of code failing (p) in any one year is 0.005 ($5 million/$1 billion)

11. Estimating SwA Effectiveness • Gartner results indicate that removing all defects would reduce the malicious attack rate by 35%* • Now assume software assurance (SwA) in the form of a red team review can cut defects by 50% • If failure p = 0.005 without SwA, then SwA can reduce the malicious attack failure by 0.005 x .35 x .5 in this example • Reduction in p due to SwA is0.000875 *Gartner reports that 35% of attacks exploit unintended vulnerabilities in code

12. Estimating Loss • Next, we need an estimate of the loss suffered if a code failure occurs • Loss could be estimated by time systems are out of service • Ship fleet costs $23M/day (development & operation) • An attack might cause a 1-day loss of service ($23M loss) • Examples across industry show higher collective losses: • Love Bug (2000) $15B damage to 3.9M systems • Code Red (2001) $1.2B damage, $740M recovery • Slammer (2002) $1B damage • Blaster (2003) $50B damage • My Doom (2004) $38B damage

13. Comparing Risk and Benefit • Avoided annual loss probability • p(reduced) = 0.000875 • Cost of loss for one day of a ship system • $25M (2006 dollars) • Annual expected loss avoided • 0.000875 x $25M = $22K • System lifetime loss avoided • 30 years x $22K = $660K (2006 dollars) • Cost of review • Can the SwA measure be done for less than $660K?

14. Nomograph of C, L, p = 0.005Assuming 30 year system life, 35% of defects lead to attacks, 50% defect reduction Area that satisfies our cost-benefit criterion

15. Private Industry Insurance Data are Not a Perfect Proxy • Firms that make up the sample drawn from the insurance database may not face the same threat and losses that the government faces • Private firms probably have different tolerance for risk than the government, and this difference influences how much coverage they buy and the price they are willing to pay • But, isn’t such a proxy better than merely guessing at p and L in those instances where the experts cannot provide estimates of the parameters?