1 / 25

Metrics Revisited

Metrics Revisited. Kim L. Jones CISM, CISSP, CRISC, MSIA. Kim L. Jones CISM, CISSP, CRISC, MSIA. Sources and Inspirations. Paul Glen, How to Speak to the Business www.leadinggeeks.com Lance Hayden, IT Security Metrics : A Practical Framework for Measuring Security & Protecting Data

akio
Download Presentation

Metrics Revisited

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Metrics Revisited Kim L. Jones CISM, CISSP, CRISC, MSIA Kim L. Jones CISM, CISSP, CRISC, MSIA

  2. Sources and Inspirations Paul Glen, How to Speak to the Business www.leadinggeeks.com Lance Hayden, IT Security Metrics: A Practical Framework for Measuring Security & Protecting Data Andrew Jaquith, Security Metrics: Replacing Fear, Uncertainty, and Doubt Kim L. Jones CISM, CISSP, CRISC, MSIA

  3. The Mantra:“Infosec is Terrible at Metrics” The metrics we can measure has little to do with security Ex: Success of Antivirus System The stuff we really need to convey is the hardest to collect/quantify “What is the sound of one hand clapping?” When we quantify numbers, they question our calculations They really don’t care about security…only compliance “What needs fixing in security, and when will it be fixed?” Kim L. Jones CISM, CISSP, CRISC, MSIA

  4. Defining the Problem Good vs. Bad Metrics Contraxioms Asking the Right Question Kim L. Jones CISM, CISSP, CRISC, MSIA

  5. Good vs. Bad Metrics Consistently Measured Cheap to Gather Technologically driven, where possible Expressed as a cardinal number or a percentage Expressed using at least one unit of measure Hours, defects, dollars, etc. Good Bad • Inconsistent Results • Expensive to Gather • Extremely Manual • Highly Subjective • High/Medium/Low Kim L. Jones CISM, CISSP, CRISC, MSIA

  6. “Contraxioms” Kim L. Jones CISM, CISSP, CRISC, MSIA

  7. Contraxiom #1 -- Work Geeks Non-Geeks • For Geeks, Work is about solving problems • Problems organize our thinking and provide a specific structure and approach • Problem solving starts in the present. • For Non-Geeks, Work is about achieving a vision • Visions are an imagined experience that get us out of bed in the morning. • Vision realization starts in the future. Kim L. Jones CISM, CISSP, CRISC, MSIA

  8. Contraxiom #1 -- Work Impact on Metrics Do we truly understand the vision? And what the business must do/is trying to do to achieve that vision? Are we relating our metrics TO the vision? This gives our metrics appropriate business context (the “So What?” factor) Kim L. Jones CISM, CISSP, CRISC, MSIA

  9. Contraxiom #6 -- Lying Geeks Non-Geeks • For Geeks, Lying is evil. Truth is sacred. • If you don’t know that it’s true, and you say it’s true, you’re lying. • Exaggerations and opinions stated as fact are lies. • For Non-Geeks, Lying is not good. Lying is bad manners • If you know that’s it’s false and say it’s true, you’re lying • Exaggerations and opinions are part of normal speech. Kim L. Jones CISM, CISSP, CRISC, MSIA

  10. Contraxiom #6 -- Lying Impact on Metrics If exaggeration is normal speech, are our “metrics” accurate or exaggerated? Business can/will ask this…after all, “spin” is natural When asked for specifics re: what will happen, are our qualifications of answers view as lack of commitment to our metrics/statements? Kim L. Jones CISM, CISSP, CRISC, MSIA

  11. Asking the Right Question Is The Road Open? • How close is the nearest rebel encampment? • Are there mines on the road? • What is the current state of rebel supplies? • Is the destination still neutral Kim L. Jones CISM, CISSP, CRISC, MSIA

  12. Asking the Right Question Are We Secure? Are We Compliant? What Is The Current Level of Risk? Are Our Controls Sufficient? Is The Risk Balanced Sufficiently To Achieve Our Vision? Kim L. Jones CISM, CISSP, CRISC, MSIA

  13. Random Thoughts… Compliance Isn’t Always Bad Testing the Hypothesis Making the Subjective Objective Data Visualization Principles Kim L. Jones CISM, CISSP, CRISC, MSIA

  14. Compliance Isn’t Always Bad Executives latch on to compliance because it meets the requirements of a good metric. The problem (as we all know) is that compliance doesn’t equal security Worse, compliance does not equalappropriately balanced risk Even if you win the metrics battle, compliance will remain an issue if you are a regulated entity Possible (useful) workaround: measuring compliance with your policy framework Meets compliance standards Sets the risk floor! Is in line with the vision! Kim L. Jones CISM, CISSP, CRISC, MSIA

  15. Testing the Hypothesis… Gathering metrics to test hypothesis can be very useful when looking to ascertain and solve problems in your network. All previous rules re: metrics, context, etc. apply Remember: don’t prove the positive…disprove the negative. Kim L. Jones CISM, CISSP, CRISC, MSIA

  16. Testing the Hypothesis… Corporate Mission: “Enable a Better Way for Trusted Commerce Infosec Mission: “We ensure the Trust in Trusted Commerce” Trust defined as: your transactions will process as expected, when expected, how expected (i.e., without alteration). Hypothesis: Our Transactions Can be Trusted Sub-Hypotheses: There are limited points of entry through which an outsider can get into our information systems Once inside, attackers cannot obtain access to internal systems because of strong passwords An intruder finding a hole somewhere cannot jump to core transactional systems Administrative credentials are difficult to obtain Kim L. Jones CISM, CISSP, CRISC, MSIA

  17. Testing the Hypothesis:Disproving the Negative • There are limited points of entry through which an outsider can get into our information systems • Attackers cannot obtain access to internal systems because of strong passwords • An intruder finding a hole somewhere cannot jump to core transactional systems • Administrative credentials are difficult for attackers to obtain • The network is porous, permitting easy access to any outsider • Attackers can obtain access to internal systems because of password policies are weak • An intruder finding a hole somewhere can easily jump straight to core transactional systems • Once on the network, attackers can easily obtain administrative credentials Kim L. Jones CISM, CISSP, CRISC, MSIA

  18. Testing the Hypothesis:Diagnostic Questions • How many sites are connected directly to the core network without intermediate firewalls? • How many sites have deployed unsecured wireless networks? • Starting with zero knowledge, how many minutes are required to gain full access to network domain controllers? • What percentage of accounts could be compromised in <15 minutes? • How many internal zones/subnets exist to compartmentalize the environment? • How many administrative-level passwords could be compromized in the same time frame? • How many universal administrator accounts exist in the environment? • The network is porous, permitting easy access to any outsider • Attackers can easily obtain access to internal systems because of password policies are weak • An intruder finding a hole somewhere can easily jump straight to core transactional systems • Once on the network, attackers can easily obtain administrative credentials. Kim L. Jones CISM, CISSP, CRISC, MSIA

  19. Making the Subjective Objective… One of the complaints re: security metrics is an inconsistency in measurement This undermines even the strongest/most significant metric as being opinion versus fact. Semi-qualitative metrics are a good starting point…but consider going a step further and implementing a standard evaluation checklist with relative values. Plotting the results of multiple assessments over a specific population may create a contextually relevant metric Kim L. Jones CISM, CISSP, CRISC, MSIA

  20. Making the Subjective Objective Kim L. Jones CISM, CISSP, CRISC, MSIA

  21. Data Visualization Principles It’s All About The Data, Not the Design Pretty designs and backgrounds are fun, but they exist to enhance the data, not overwhelm it Simple Is Better Erase what you don’t need Avoid 3-D Hint: Wizards aren’t necessarily helpful Simplify the Color Palette Muted, Primary Colors Kim L. Jones CISM, CISSP, CRISC, MSIA

  22. Data Visualization Principles Label Honestly and Accurately Titles should be meaningful Labels should enhance understanding Always identify units of measure Avoid clutter Consider the Best Depiction of Data Pie Chart? Stacked Bar? Pareto? Test the Data! Grant’s Captain Kim L. Jones CISM, CISSP, CRISC, MSIA

  23. Wrapping it Up… Security is, at a fundamental level, a state of mind Ditto for balanced risk It stands to reason, then, that measuring security and/or risk can be like catching a moonbeam “What is the sound of one hand clapping?” Metrics and measurement are both art and science…you need to study both Make your metrics contextually relevant What’s the vision? Be sure you’re answering the right question!! Kim L. Jones CISM, CISSP, CRISC, MSIA

  24. Kim L. Jones CISM, CISSP, CRISC, MSIA Questions?

  25. Contact Data… Kim L. Jones CISM, CISSP, CRISC, MSIA (480) 253-9120 Kljones.cism@gmail.com Kim L. Jones CISM, CISSP, CRISC, MSIA

More Related