1 / 56

C2 Maturity Model Experimental Validation

C2 Maturity Model Experimental Validation. Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts. Agenda. Background Available Data Analysis Plan Results of Analyses Conclusions. Background. SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability

alaina
Download Presentation

C2 Maturity Model Experimental Validation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. C2 Maturity Model Experimental Validation Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts

  2. Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions

  3. Background • SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability • SAS-065 is engaged in a variety of activities to “validate” this model • Case studies: Analysis of historical complex endeavors to establish construct validity • Analysis of Experimentation Data to test a set of maturity model hypotheses • ELICIT is an experimentation platform that instruments the actions of a group of participants engaged in a situational awareness problem • ELICIT experiments have explored differences between “edge” and “hierarchical” organizations

  4. ELICIT Scenario Hierarchy Edge • The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack • Participants can share factoids directly with each other or post factoids to websites • Participants build awareness by gathering and analyzing factoids and interacting with one another • No one is given sufficient information to solve their assigned problem without receiving information from others • The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained • C2 approach for this series of experiments were designated prior to the start of the run asHierarchy or Edge

  5. Hypotheses • “Hierarchical” organizations as instantiated in ELICIT experiments correspond to De-Conflicted C2 in the NATO C2 Maturity Model • “Edge” organizations as instantiated in ELICIT experiments correspond to a higher level of maturity in the NATO C2 Maturity Model • Hypotheses: • Edge organizations exhibit the behaviors associated with Coordinated and Collaborative levels of C2 Maturity • Hierarchical organizations exhibit the behaviors associated with the De-conflicted level of C2 Maturity • Edge organizations exhibit higher levels of effectiveness and efficiency than Hierarchical organizations

  6. ELICIT Data Set • Includes data from 37 ELICIT experimentation trials • Venues • Boston Univ. (2 runs) • NPS (16 runs) • Portugal (6 runs) • USMA (3 runs) • Singapore (10 runs) • Organization Types • 18 Edge • 19 Hierarchy

  7. Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions

  8. Variables of Interest C2 Maturity Level Allocation of Decision Rights Patterns of Interaction Network Characteristics & Performance Distribution of Information Shared Understanding Shared Information Shared Awareness Task Performance Info Sharing & Collaborative Behaviors Quality of Information Quality of Awareness Quality of Understanding Measures of Merit Task Difficulty Individual & Team Characteristics Legend Controllable Quality of Information Sources Partially Controllable Culture

  9. Dependent Variables • MOE = Quality of Awareness and Shared Awareness • Correctness (Authorized Correct IDs) • Timeliness (Person-Minutes with Correct IDs) • Accuracy rate (Correct IDs/Total IDs) • Efficiency, Given Effectiveness • Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) • Speed (Time of Earliest Correct ID) • Agility • Effectiveness over problem difficulty

  10. C2 Approach Independent Variables • Hierarchy v. Edge • We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level • Each run will be mapped to a point in the C2 Approach Space based on observed behaviors • Rules of Interaction • Website access • Sharing permissions • Initial Distribution of Factoids • Invariant in existing runs

  11. C2 Approach Intervening Variables • Patterns of Interaction • Characteristic path length • Clustering coefficient • Connectedness • Distribution of Information • The average number of unique facts to which each participant has access as a function of time

  12. Measures of C2 Effectiveness (MOCE) • Quality of Information Position • Percentage of relevant facts for the assigned task that a participant can access as a function of time • Percentage of key facts for the assigned task that a participant can access as a function of time • Extent of Shared Information • The average number of participants that have access to each fact as a function of time • The average number of participants that have access to each key fact as a function of time

  13. Intervening Behavioral Variables • Activity over time (sharing, website posts, website pulls, ID attempts) • Sharing • Peer-to-peer sharing • Posting • Information Seeking • Pulling • Identification Attempts

  14. Other Independent Variables • IDs allowed • ELICIT experience of the player • Factoid set (problem difficulty) • Translated factoids v. original • Native Language (English v. Other) • Communications media • Postcards • Chat • Time available • Degree of Education (Graduate, Undergraduate) • Seniority (Rank) • Subcultures (Military, Civilian, Special Forces, Civil Servants)

  15. Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions

  16. Dependent Variables • MOE = Quality of Awareness and Shared Awareness • Correctness (Authorized Correct IDs) • Timeliness (Person-Minutes with Correct IDs) • Accuracy rate (Correct IDs/Total IDs) • Efficiency, Given Effectiveness • Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) • Speed (Time of Earliest Correct ID) • Agility • Effectiveness over problem difficulty

  17. MOE: CorrectnessFraction of Participants with Correct IDs Edge Hierarchy

  18. MOE: CorrectnessFraction of Participants with Correct IDs Edge Better Than HierarchySignificant at 0.95 level

  19. MOE: CorrectnessFraction of Participants with Authorized Correct IDs Edge Hierarchy Note: Half credit given for partially correct answers in authorized areas

  20. MOE: CorrectnessFraction of Authorized Participants with Correct ID Edge Better Than HierarchySignificant at 0.95 level |t-Ratio| = 7.46

  21. Measuring Timeliness:Person-Minutes with Correct ID • In the ELICIT scenario, operational value is associated both with the number of participants achieving the correct answer, as well as when they achieved it • Person minutes with correct ID captures the proportion of possible situational understanding achieved by trial participants over the time of the trial • For each minute, each participant’s level of understanding is assessed by the score assigned to his/her most recent identification attempt • This value is summed over the duration of the trial to arrive at the person-minutes correct for each participant • Person-minutes correct for the trial is calculated by summing over all participants • A ratio of the person minutes correct for the trial to the total person minutes available in the trial adjusts for varying trial lengths

  22. MOE: TimelinessFraction of Person-Minutes Correct Edge Hierarchy

  23. MOE: TimelinessFraction of Person-Minutes Correct Edge Better Than HierarchySignificant at 0.95 level

  24. MOE: Accuracy RateFraction of ID Attempts that are Correct Edge Hierarchy

  25. MOE: Accuracy RateFraction of ID Attempts that are Correct Edge Better Than HierarchySignificant at 0.95 level

  26. Efficiency: Productivity (Actions)Correct ID / Total Actions Edge Hierarchy

  27. Efficiency: Productivity (Actions)Correct ID / Total Actions Difference not Significantat 0.95 level |t-Ratio| = 1.41

  28. Efficiency: Productivity (Person-Minutes)Correct IDs / Person-Minutes Available Edge Hierarchy

  29. Efficiency: Productivity (Person-Minutes)Correct IDs / Person-Minutes Available Edge Better Than HierarchySignificant at 0.95 level

  30. Efficiency: SpeedTime of Earliest Correct ID Edge Hierarchy

  31. Efficiency: SpeedTime of Earliest Correct ID Difference not Significant at 0.95 level |t-Ratio| = 1.62

  32. AgilityMOE: Timeliness over Problem Difficulty Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct DifficultFactoidSets Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct StandardFactoidSets Edge Hierarchy Edge Effectiveness is Degraded Less Than Hierarchy

  33. C2 Approach Intervening Variables • Patterns of Interaction • Characteristic path length • Clustering coefficient • Connectedness • Distribution of Information • The average number of unique facts to which each participant has access as a function of time

  34. Distribution of InformationThe cumulative number of unique facts to which each participant has access over time Time (minutes)

  35. Distribution of InformationThe cumulative number of unique facts to which each participant has access over time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)

  36. Measures of C2 Effectiveness • Quality of Information Position • Percentage of relevant facts for the assigned task that a participant can access as a function of time • Percentage of key facts for the assigned task that a participant can access as a function of time • Extent of Shared Information • The average number of participants that have access to each fact as a function of time • The average number of participants that have access to each key fact as a function of time

  37. MOCE: Quality of Information PositionPercentage of relevant facts for the assigned task that a participant can access as a function of time Time (minutes)

  38. MOCE: Quality of Information PositionPercentage of relevant facts for the assigned task that a participant can access as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)

  39. MOCE: Quality of Information PositionPercentage of key facts for the assigned task that a participant can access as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

  40. MOCE: Quality of Information PositionPercentage of key facts for the assigned task that a participant can access as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

  41. MOCE: Extent of Shared InformationThe average number of participants that have access to each fact as a function of time Time (minutes)

  42. MOCE: Extent of Shared InformationThe average number of participants that have access to each fact as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)

  43. MOCE: Extent of Shared InformationThe average number of participants that have access to each key fact as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

  44. MOCE: Extent of Shared InformationThe average number of participants that have access to each key fact as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

  45. Intervening Behavioral Variables • Activity over time (sharing, website posts, website pulls, ID) • Sharing • Peer-to-peer • Posting • Information Seeking • Pulling • Identification

  46. Activity over Time Peer to Peer Sharing Time (minutes)

  47. Activity over Time Peer to Peer Sharing Hierarchy Better Than Edge at 10 minNo Significant Difference at 45 min(at 0.95 level) Time (minutes)

  48. Activity over Time Posts Time (minutes)

  49. Activity over Time Posts No Significant Differenceat 0.95 level Time (minutes)

  50. Activity over Time Information Seeking (Pulls) Time (minutes)

More Related