DoD
Download
1 / 43

Ver 10.0 September 29, 2005 - PowerPoint PPT Presentation


  • 177 Views
  • Uploaded on

DoD Counterintelligence STRATEGY MAPPING. William L. McCoy Senior Program Manager Lockheed Martin Integrated Technology. Ver 10.0 September 29, 2005. STRATEGY MAPPING OBJECTIVE.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Ver 10.0 September 29, 2005' - Pat_Xavi


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Briefing overview

DoD

Counterintelligence

STRATEGY

MAPPING

William L. McCoy

Senior Program Manager

Lockheed Martin Integrated Technology

Ver 10.0 September 29, 2005


Briefing overview

STRATEGY MAPPING OBJECTIVE

To develop a methodology that defensibly and quantitatively maps the DoD CI Strategy (and eventually DoD CI resources to achieve it) to DNI Strategic Objectives, NCIX Pillars, Defense Intelligence Goals, and other guidance documents.


Briefing overview
BRIEFING OVERVIEW

  • Purpose:Recommend survey universe, sample size (if required)

    and analysis methodology for strategic mapping project that maps

    DoD CI General Goals to:

    • National Counterintelligence Pillars

    • Director of National Intelligence Strategic Objectives

    • USD(I) Defense Intelligence Goals

  • Discussion:

    • DT Survey Results

    • Part 1 – Universe and Sample Size Methodology

      • Recommendation and Decision

    • Part 2 – Data Reduction and Analysis

      • Recommendation and Decision

  • Next Steps



Briefing overview

PART 1

UNIVERSE AND SAMPLE SIZE METHODOLOGY


Briefing overview

SURVEYING

  • Survey types:

    • Sample – more involved as we must make corrections

      for the error introduced by not surveying the universe.

      Sampling is required if the universe is too large.

    • Population – more straight forward as no errors are

      introduced by sampling but requires manageable universe.

  • Sampling introduces an additional step of determining and

  • correcting for the variance – hence the arcane statistics.

The analysis commences after the numbers are crunched.

The results can be accepted at face value or adjusted/revised

based on subject knowledge.


Briefing overview

OPTION DISCRIMINATORS

OPERATIONAL DEFINITIONS FOR SURVEY

“TERMS OF ART” DISCRIMINATORS


Briefing overview

UNIVERSE ASSUMPTIONS

  • OPTION 3

  • estimate

  • CIRROC

  • Member

  • Organizations 80

  • Total 80

  • OPTION 2

  • estimates

  • AF OSI 65

  • Navy NCIS 75

  • USMC 25

  • Army 55

  • Agencies 80

  • Total 300

  • OPTION 1

  • estimates

  • AF OSI 450

  • Navy NCIS 700

  • USMC 200

  • Army 350

  • Agencies 300

  • Total 2,000

  • SAMPLE SIZE

  • Target population of 2,000

  • Confidence interval +/- 5%

  • Confidence level of .95

    • Then sample size = 92

    • SAMPLE SIZE

    • Target population of 300

  • Confidence interval +/- 5%

  • Confidence level of .95

    • Then sample size = 73

    • SAMPLE SIZE

      • Target population of80

      • 100% survey possible

    • Then target universe = 100%

    ASSUMPTIONS

    OPTION 1: The survey is administered to randomly selected DoD CI leadership and planners down to

    unit officersand senior NCOs and includes civilians.

    OPTION 2: Uses OPTION 1 assumption but restricts survey participation to DoD CI leadership, planners,

    and resource managers down to major unit level.

    OPTION 3: Limits the survey to National level DoD CI leaders, strategic planners, and resource managers.


    Briefing overview

    OPTION 1 DETAIL

    Assumption: To map CI strategy we need input from all levels of the DoD CI community (OSD, CIFA, Services,

    Agencies, Regions, Brigades, Groups, Battalions, Detachments, etc.) from DoD CI leadership and planners

    down to unit CI officers and senior CI NCOs and civilian equivalents.

    • PROs:

      • Elicits input from those not normally associated with strategy issues.

      • Provides an operator perspective on CI strategy.

    • CONs:

      • Partially substitutes opinion for judgment.

      • Includes population with out experience with or insight into National or Defense CI activities.

      • Risk of missing significant information – may lower the sample mean below the threshold.


    Briefing overview

    OPTION 2 DETAIL

    Assumption: To map the strategy we need to know the view of all headquarters elements (less support) in the DoD CI

    community from DoD CI leadership, and planners, CI officers and senior CI NCOs and civilian equivalents at OSD, CIFA,

    Services, and Agencies down to Regions and Brigades.

    • PROs:

      • Provides perspectives from those dealing with both CI planning and execution issues.

      • Develops a broadly based input on strategy by including more operational and very strategic CI

        leaders and planners.

        CONs:

      • Population may not be conversant or experienced in National or Defense strategy issues.

      • Includes population with out experience with or insight into National or Defense CI activities.

      • Risk of missing significant information – may lower the sample mean below the threshold.


    Briefing overview

    OPTION 3 DETAIL

    Assumption: To map the strategy we need to know the perspective of DoD level leaders, strategy planners, and resource managers within the DoD CI community (OSD, CIFA, Services, Agencies).

    • PROs:

      • Generates responses from a population with ongoing knowledge of and experience with

        National and Departmental strategy, budget, and resource issues.

      • Develops a broadly based view of strategy from DoD CI organizations.

        CONs:

      • Survey population includes only headquarters leadership – misses CI field operator challenges.


    Briefing overview

    RECOMMENDATION:

    OPTION 3

    • The most relevant population –

    • current knowledge of and engaged

    • in National and Departmental CI

    • goals, strategies, resources, and

    • budgets.

    • Responders would have the

    • background and current knowledge

    • relevant to measuring strategy linkage.

    • Less risk of missing significant

    • results as there is no sample, thus no

    • risk of Type 2 error.


    Briefing overview

    PART 2

    DATA REDUCTION AND ANALYSIS



    Briefing overview

    TAKING THE TOP SCORE

    OPTION 1

    Results from initial DT survey.

    SELECTING LINKAGE DETERMINED BY: THE TOP ONE OR TWO

    RESULTS IN A CATEGORY

    PRO: Easy to understand.

    CON: Does not explain why the top one or two cut-off; why

    not the top three or four. More robust approaches

    possible.


    Briefing overview

    ADJUSTING THE CUTOFF BY THE

    STANDARD DEVIATION

    OPTION 2

    Results from initial DT survey.

    SELECTING LINKAGE DETERMINED BY:USING THE MEAN PLUS

    ONE STANDARD DEVIATION

    PRO: Still easy concept to understand and answers the

    “why” for the cut-off.

    CON: 1 sigma is too high for including alternates and splitting a

    sigma, while doable, is questionable.


    Briefing overview

    CUTOFF USING THE 85TH PERCENTILE

    OPTION 3

    Results from initial DT survey.

    SELECTING LINKAGE DETERMINED BY:USING PERCENTILE

    PRO: Understandable concept, answers the “why” for the cut-off. the range can

    easily be broadened to include secondary selections by adjusting the

    percentile.In some cases the Percentile approach may “table up”

    additional choicesnot presented by OPTION 1.

    CON: Less relationship to normal distribution theory.


    Briefing overview

    RECOMMENDATION

    OPTION 3 is recommended – it provides

    a defensible solution with reasonable rigor

    using commonly understood data analysis

    techniques.


    Next steps
    NEXT STEPS

    • Develop survey instructions and invitations according

      to the approved universe – <1 week

    • Administer the survey – 1 week

    • Complete data steps:

      • Input – 3 days

      • Reduction – 1 day

      • Analysis – 1 day

    • Publish, disseminate, and apply results




    Briefing overview

    EXCURSION –

    TWO BASIC TYPES OF SURVEYS

    Surveys are divided into two categories:

    • 1. A survey of the universe – also called a “census” or a “poll”

    • 2. A sample survey that is representative of the universe:

      • The only time we don’t survey the universe is when time, cost, or

    • accessibility of respondents create the need for a sample – in other

    • words, when we have no other choice.

      • The goal of a sample is to produce the same results that would have been obtained had every single member of a universe been

    • interviewed.

    • The key to reaching this goal is a fundamental principle called equal probability of selection.

    • Cluster, stratified, and other forms of surveys are based on the

    • foregoing.


    Briefing overview

    TO DETERMINE THE UNIVERSE,

    “WHO CAN TELL US WHAT WE NEED TO KNOW?”

    UNIVERSE SELECTION PRINCIPLES:

    • The survey's universe must fit the facts of the case.

    • The target population must correspond to the topic studied.

    • A universe which is relevant to the problem being studied and

    • includes respondent qualification requirements is a vital

    • requirement of high quality research.

    • When selecting the universe every member of that universe

    • must have an equal probability of sample selection.

    INDICATE THE UNIVERSE SHOULD BE POPULATED WITH:

    • Those in DoD who are engaged and familiar with the National and Departmental CI goals, strategies, resources, and budgets.


    Briefing overview

    HYPOTHESIS TESTING:

    STEP 2 EXPLANATION

    Setting the Level of Significance

    • The significance level is used for accepting or rejecting the null hypothesis.

    • The difference between the results of the experiment and the null hypothesis

    • is determined.

    • Assuming the null hypothesis is true, the probability of a difference that large

    • or larger is computed .

    • This probability is compared to the significance level.

    • If the probability is less than or equal to the significance level, then the null

    • hypothesis is rejected and the outcome is said to be statistically significant.

    • The lower the significance level, the more the data must diverge from the

    • null hypothesis to be significant.


    Briefing overview

    FORMULA FOR CALCULATING SAMPLE SIZE

    Sample Size

    Z2 * (p) * (1-p)

    SS = -----------------------

    c2

    1.962 * (.5) * (1-.5)

    SS = -----------------------

    .102

    1.962 * (.5) * (1-.5)

    SS = -----------------------

    .102

    .964

    SS = -----------------------

    .01

    Where:

    SS = 96 rounded

    More follows


    Briefing overview

    . . . AND CORRECTING FOR A FINITE UNIVERSE

    SS

    Corrected SS = -----------------------

    SS-1

    1+ ------

    pop

    96.04

    Corrected SS = -----------------------

    96.04-1

    1+ ----------

    2,000

    where:

    96.04

    Corrected SS = -----------------------

    95.04

    1+ ----------

    2,000

    96.04

    Corrected SS = -----------------------

    1.04752

    Corrected SS = 92 rounded


    Briefing overview

    . . . OR YOU CAN USE THE FREEWARE

    SAMPLE SIZE CALULATOR

    Given a universe of 2000 and a confidence

    interval of 10 (+\- 5), you are 95% certain

    that a sample size of 92 will provide results

    consistent with a total survey of the universe.

    WHICH SAYS:


    Briefing overview

    STRATEGY MAPPING

    SAMPLE SIZE METHODOLOGY

    EXAMPLE

    • Assuming:

    • DoD CI strength of 2,000

    • Confidence interval of +/- 5%

    • Confidence level of .95

    • Then sample size = 92

    }

    FYI:

    The sample

    size for a

    population

    of 3,000 is

    93 and for

    300 is 73.

    Z2 * (p) * (1-p)

    SS = -----------------------

    c2

    SS

    Corrected SS = -----------------------

    SS-1

    1+ ------

    pop

    BASED ON

    OR



    Briefing overview

    RESEARCH PROBLEM

    • The most difficult task in developing a research project is to narrow

    • down the field of study and the research problem. A distinction needs

    • to be made between a problem and a research problem.

    • A problem is an observed discrepancy or gap between what is known

    • and not known.

    • The identification of the problem is an interpretation of the gap based

    • on a set of observations.

    • A research problem is a judgment drawn from the interpretation of the gap.


    Briefing overview

    RELIABILITY AND VALIDITY

    VALIDITY: Information is presented or used in the

    way for which it was intended.

    RELIABILITY: We can expect to obtain the same

    information time after time.


    Briefing overview

    DATA MANAGEMENT

    SURVEY

    INPUT

    STORAGE

    Data summed by guidance

    Category and by General and

    Performance Goal, standardized

    to 100, and reviewed through

    correlation and other techniques.

    Summary or unsummarized data

    provided to CI leadership and

    strategy planners for review and

    analysis.

    COMMENCE ANALYSIS


    Briefing overview

    The standard error of a sample of sample size is the sample's

    standard deviation divided by the square root of n. It therefore

    estimates the standard deviation of the sample mean based on

    the population mean. Note that while this definition

    makes no reference to a normal distribution, many uses of this

    quantity implicitly assume such a distribution.

    The standard error of an estimate may also be defined as the

    square root of the estimated error variance of the quantity,


    Briefing overview

    RESEARCH QUESTION

    How does DoD counterintelligence strategy

    map into national and DoD intelligence and

    counterintelligence guidance?


    Briefing overview

    HYPOTHESIS TESTING:

    The Null and Alternative Hypotheses

    H0 – NULL HYPOTHESIS: There is no relationship between DoD

    Counterintelligence Strategy and National Counterintelligence Pillars.

    H0: µ - M < 0

    H1 –ALTERNATIVE HYPOTHESIS: There is a relationship between

    DoD Counterintelligence Strategy and National Counterintelligence

    Pillars.

    H1: µ - M =>0


    Briefing overview

    HYPOTHESIS TESTING:

    • Next steps:

    • Set the Level of Significance at .95; alpha = .05

    • Identify the Test Statistic and calculate the

    • critical value (or probability)

    • Formulate the decision rule - in this case if the

    • Test Statistic is => 0 reject the Null Hypothesis


    Briefing overview

    DATA REDUCTION AND ANALYSIS

    DoD 2

    DoD 4

    DoD 6


    Briefing overview

    HYPOTHESIS TESTING

    Initial DT NCI Survey Results


    Briefing overview

    HYPOTHESIS TESTING

    Problem with Sampling

    Fairly robust statistics required to understand the survey result. It could have

    been poor universe selection or an improperly designed survey instrument.


    Briefing overview

    HYPOTHESIS TESTING

    Initial DT NCI Survey Results


    Briefing overview

    HYPOTHESIS TESTING

    Problem with Sampling

    Fairly robust statistics required to understand the survey result. It could have

    been poor universe selection or an improperly designed survey instrument.


    Briefing overview

    SELECTION WITHOUT REPLACEMENT

    GOAL: Provide an NCI Pillar for each DoD CI Strategy General Goal

    • Assign the DoD CI GG having the highest score

    • If the DoD CI GG will be repeated, select the

    • second highest DoD CI GG score

    • Continue until all DoD CI GGs are exhausted


    Briefing overview

    OPINION VICE JUDGMENT

    opinion  u'pinyun

    A personalbelief or judgment that is not founded on proof or certainty

    A belief or sentimentshared by most people; the voice of the people

    A messageexpressing a belief about something; the expression of a belief that is heldwith

    confidence but not substantiated by positiveknowledge or proof

    The legal documentstating the reasons for a judicial decision

    The reason for a court's judgment (as opposed to the decision itself)

    A vagueidea in which some confidence is placed