Download

Intelligent Approaches to Lessons Learned Processes






Advertisement
/ 67 []
Download Presentation
Comments
alia
From:
|  
(1118) |   (0) |   (0)
Views: 70 | Added:
Rate Presentation: 0 0
Description:
Intelligent Approaches to Lessons Learned Processes. Rosina O Weber University of Wyoming Navy Center for Applied Research in AI, Naval Research Lab. Intelligent Approaches to Lessons Learned Processes. Collaborators: David W. Aha Hector Munoz Avila Len Breslow Nabil Sandhu. Outline.
Intelligent Approaches to Lessons Learned Processes

An Image/Link below is provided (as is) to

Download Policy: Content on the Website is provided to you AS IS for your information and personal use only and may not be sold or licensed nor shared on other sites. SlideServe reserves the right to change this policy at anytime. While downloading, If for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.











- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -




Slide 1

Intelligent Approaches to Lessons Learned Processes

Rosina O Weber

University of Wyoming

Navy Center for Applied Research in AI, Naval Research Lab

Slide 2

Intelligent Approaches to Lessons Learned Processes

Collaborators:

David W. Aha

Hector Munoz Avila

Len Breslow

Nabil Sandhu

R.O.Weber Calgary 3 Aug 2001

Slide 3

Outline

  • Introduction

    • Context

  • Problems with lessons learned systems

  • Methodology

  • Monitored Distribution

  • Case Representation

  • Lesson Elicitation Tool

  • Next Steps

R.O.Weber Calgary 3 Aug 2001

Slide 4

Context

  • Knowledge management context

  • Lessons learned systems (LLS)

  • Organizations adopting LLS

  • Lessons learned definition, representation and example

  • Lessons learned process

R.O.Weber Calgary 3 Aug 2001

Slide 5

Knowledge management context

  • Three types of KM initiatives*

    • knowledge repositories

    • knowledge access and transfer

    • knowledge environment

  • Knowledge repositories

    • Internet

    • industry oriented (alert systems)

    • organization oriented (lessons learned systems)

R.O.Weber Calgary 3 Aug 2001

5

Slide 6

KNOWLEDGE

ARTIFACTS

Lessons learned systems

  • Lessons learned systems are knowledge repositories of knowledge artifacts

  • Examples of knowledge artifacts are lessons, alerts, best practices, reports, video clips, etc.

R.O.Weber Calgary 3 Aug 2001

6

Slide 7

Lessons learned systems in organizations

  • Aha & Weber (Eds.)Intelligent lessons learned Systems. Papers from the AAAI 2000 Workshop(Technical Report WS-00-03) AAAI Press

  • Weber, Aha & Becerra-Fernandez (survey) Intelligent lessons learned Systems. 2001 International Journal of Expert Systems Research & Applications, Vol. 20, No. 1., 17-34.

R.O.Weber Calgary 3 Aug 2001

7

Slide 8

government

Construction Industry Inst.

Honeywell

GM

Hewllet Packard

Bechtel Jacobs Company

Lockheed Martin E. Sys, Inc

DynMcDermott Petroleum Co.

Xerox

IBM

BestBuy

Siemens

int’l

US

European Space Agency

Italian (Alenia)

French (CNES)

Japanese (NASDA)

United Nations

Air Force

Army

Coast Guard

Joint Forces

Marine Corps

Navy

int’l

US

Department of Energy: SELLS

NASA (Ames, Goddard)

Canadian Army Lessons Learned Centre

non-government

non-military

military

8

Slide 9

Lessons learned definition…

…or organizational lessons, lessons, lessons identified

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

R.O.Weber Calgary 3 Aug 2001

9

Slide 10

Lessons learned example

applicable task

Installing custom stereo speakers.

conditions for applicability

The car is the Porsche Boxster.

lesson suggestion

Make sure you distinguish the wires leading to the speakers from the wires leading to the side airbag.

Rationale

Somebody has cut the wrong wire because they look alike and the airbag went off with explosive force. This means spending several thousand dollars to replace the airbag in addition to be a potential hazard.

From article “Learning from Mistakes” about Best Buy in Knowledge management magazine, April 2001.

R.O.Weber Calgary 3 Aug 2001

11

Slide 11

Lessons learned process

R.O.Weber Calgary 3 Aug 2001

12

Slide 12

Lessons distribution sub process

R.O.Weber Calgary 3 Aug 2001

20

Slide 13

Lesson distribution methods

Broadcasting

bulletins, doctrine

Passive

Standalone repository

Active casting

list servers,

information gathering tools

Pull

Push

R.O.Weber Calgary 3 Aug 2001

20

Slide 14

Problems with lesson distribution methods

  • Distribution is divorced from targeted organizational processes.

  • Users maynot know or be reminded of the repository, as they need to access a standalone tool to search for lessons.

  • Users maynot beconvinced of the potential utility of lessons.

  • Users maynot have the time and skills to retrieve and interpret relevant lessons.

  • Users maynotbe able to apply lessons successfully.

R.O.Weber Calgary 3 Aug 2001

21

Slide 15

Organization’s

members

Repository of lessons learned

Here is a gap

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 16

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 17

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 18

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 19

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 20

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 21

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 22

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 23

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 24

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 25

Organization’s

members

Repository of lessons learned

Monitored distribution

Organizational processes

R.O.Weber Calgary 3 Aug 2001

Slide 26

Organization’s

members

Repository of lessons learned

Monitored distribution

Lesson repository is integrated with targeted processes

…and lessons aredistributed when and where they are needed.

Organizational processes

R.O.Weber Calgary 3 Aug 2001

22

Slide 27

Problems with lessons learned systems

  • Technological

  • Human

  • Managerial

R.O.Weber Calgary 3 Aug 2001

13

Slide 28

Technological issues

Standalone distribution outside context of reuse

Lessons disseminated in context of reuse

Low precision and recall in text databases

Case retrieval for lesson dissemination

  • cases indexed by applicability

    Convert lessons into cases/Collect cases

    Requirement: machine recognizable format

    Textual representation of lessons

    Case strutcture for lessons

R.O.Weber Calgary 3 Aug 2001

14

Slide 29

Human issues

Lesson authors

Lesson validators

Lesson (re)users

  • Lack of training/instructions: content and format

  • Hard to validate textual descriptions

  • Have to access the repository in another context

  • Have to accept the potential benefit of lessons

  • Have the skills to search for lessons

  • Have to interpret textual lessons

R.O.Weber Calgary 3 Aug 2001

15

Slide 30

Managerial issues

  • Determine, communicate and enforce standards for lesson collection and representation

  • Define structured format

  • Embed knowledge in targeted processes

  • Monitor knowledge transfer

  • Oversee knowledge reuse

Knowledge collection

Knowledge validation

Knowledge reuse

R.O.Weber Calgary 3 Aug 2001

16

Slide 31

Monitored distribution

Case retrieval

Lessons as structured cases

Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format

Technological, Human, Managerial issues

Standalone repository

Retrieval method

Textual format of lessons

Collection method

embed instructions

for lesson submission

R.O.Weber Calgary 3 Aug 2001

17

Slide 32

Methodology

Prescriptive KM infra structure for frameworks

human users

  • Monitored distribution

  • Case representation

  • Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format

LET

organizational processes

lesson repository for

monitored distribution

Case base

case

of lessons

base

R.O.Weber Calgary 3 Aug 2001

18

Slide 33

Frameworks

  • Monitored Distribution

  • Case represebtation

  • Lesson Elicitation Tool

R.O.Weber Calgary 3 Aug 2001

19

Slide 34

Monitored Distribution

Slide 35

Problems & Solutions

Lessons are integrated to targeted organizational processes.

Users don’t need to be reminded of the repository because they don’t need to access a standalone tool.

No additional time or skills are required.

Case retrieval of disambiguated knowledge increase recall and precision

Whenever possible, an ‘apply’ button allows the lesson to be automatically executable.

Distribution is divorced from targeted organizational processes.

Users maynot be reminded of the repository, as they need to access a standalone tool to search for lessons.

Users maynot have the time and skills to retrieve relevant lessons.

Text databases have low levels of precision and recall

Users maynotbe able to apply lessons successfully.

20

Slide 36

Monitored distribution characteristics

  • Distribution is tightly integrated to the targeted processes and distribute lessons when and wherethey are needed.

  • Represent lessons as form-like cases.

  • Distribute lessons using case-retrieval/ and retrieve lessons based on similarity.

  • Additional benefits are:

  • Case representation facilitates interpretation.

  • Users access the lesson rationale to evaluate its potential utility.

  • Cases are retrieved in the context of similar experiences.

R.O.Weber Calgary 3 Aug 2001

21

Slide 37

Noncombatant Evacuation Operations:

Military operations to evacuate noncombatants whose lives are in danger and rescue them to a safe haven

R.O.Weber Calgary 3 Aug 2001

Slide 38

Assembly Point

Campaign headquarters

Intermediate Staging Base

.

safe haven

NEO site

Slide 39

Example in HICAP

  • HICAP is a plan authoring tool suite http://www.aic.nrl.navy.mil/hicap

  • Muñoz-Avila et al., 1999

  • Users interact with HICAP by refining an HTN (hierarchical task network) through decompositions

R.O.Weber Calgary 3 Aug 2001

24

Slide 40

safe haven

NEO site

R.O.Weber Calgary 3 Aug 2001

Slide 41

Selecting the Suggested Case…

R.O.Weber Calgary 3 Aug 2001

Slide 42

Expanding yields…

R.O.Weber Calgary 3 Aug 2001

Slide 43

And the user is notified of a lesson

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing them.

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing them.

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing

WHY: The enemy might be able to infer that SOF are involved, exposing

R.O.Weber Calgary 3 Aug 2001

Slide 44

After applying the lesson

R.O.Weber Calgary 3 Aug 2001

Slide 45

Evaluation

  • Hypothesis

    • Using lessons will improve plan quality

  • Methodology

    • Simulated HICAP users generated NEO plans with and without lessons

    • NEO executor implemented plans

      • Plan total duration

      • Plan duration before medical assistance

      • Casualties: evacuees, FF, enemies

R.O.Weber Calgary 3 Aug 2001

30

Slide 46

Plan evaluator

  • non deterministic (100 plans 10 times each)

  • 30 variables: 12 random

  • length of plans 18 steps

  • size of planning space 3,000,000

  • 13 lessons

  • Actions and their influences

R.O.Weber Calgary 3 Aug 2001

31

Slide 47

Plan evaluator: actions

  • Plans where evacuees were transported by land modes have an increased chance of being attacked by enemies.

  • Plans that combined weather with too strong wings have a small chance of helicopter crash.

  • When an attack or crash happens it increases the number of casualties among evacuees and FF (in proportion to # of evacuees).

R.O.Weber Calgary 3 Aug 2001

32

Slide 48

Plan evaluator: lessons

Conditions for applicability:

There are representatives of different branches assigned to participate.

Lesson suggestion:

Assign representatives of all forces to plan.

Rationale:Lack of representatives prevent good communication causing delays and miscommunication.

  • Conditions for applicability:

  • There are hundreds or more evacuees as to justify a security effort.

  • Lesson suggestion:

  • Assign EOD personnel.

  • Rationale:An evacuee once asked what to do with their weapons.

R.O.Weber Calgary 3 Aug 2001

33

Slide 49

Results

*The resulting values are averages

no lessons

with lessons

reduction

NEO plan

total duration*

32h48

18 %

39h50

duration until

medical assistance*

24h13

18 %

29h37

casualties

among evacuees

24 %

11.48

8.69

casualties among

friendly forces

6.57

30 %

9.41

casualties

among enemies

-2 %

3.08

3.14

R.O.Weber Calgary 3 Aug 2001

34

Slide 50

Case representation

Slide 51

CBR Cycle and Knowledge Processes

distribute

store

New

knowledge

collection

verify

Slide 52

Case Representation

case problem  indexing elements

case solution  reuse elements

Distributing process knowledge in the context of the targeted process.

R.O.Weber Calgary 3 Aug 2001

36

Slide 53

Lessons learned representation

  • indexing elements:

    • applicable task

    • conditions for applicability

  • reuse elements:

    • lesson suggestion

    • rationale

  • indexing elements:

    • applicable task

    • conditions for applicability

  • reuse elements:

  • indexing elements:

  • reuse elements:

R.O.Weber Calgary 3 Aug 2001

37

Slide 54

Problems & Solutions

If more than one lesson happens to be retrieved, the representation allows the user to assess its relevance immediately

Pre-defined content and format limit the content to what is applicable, facilitating correct use

The conversion into cases forces disambiguation facilitating interpretation

Structured representation can potentially allow automatic verifications, without adding unnecessary content

Users maynotbe able to correctly interpret retrieved lessons because they may be long and not well written

Free text collection allows unlimited content and have the potential to cause misuse of lessons

Users may not be able to interpret long and ambiguous texts

Textual representations complicate validation, sometimes validation results in more text added

38

Slide 55

Lesson Elicitation Tool

Slide 56

Elicitation Tool

  • LET is supported by generic and domain-specific lexicons of expressions and verbs (do not store unless relevant)

  • Confirmations search for clues indicating user’s need for help

  • A domain lexicon supports disambiguation at run-time Uses a subset of NL based on the case representation by using a template-based elicitation with pre-defined grammar structures thus avoiding NL parsing

  • LET supports conversation to acquire new concepts for the ontology

R.O.Weber Calgary 3 Aug 2001

39

Slide 57

1.2 Fields Elicitation

1.2.1 APPLICABLE ACTION Elicitation

1.2.2 SUGGESTION Elicitation

1.2.3 CONDITIONS Elicitation

1.2.4 ORIGINATING EVENT Elicitation

1.2.5 View text (edit)

1.2.a Word comparison

1.2.b Vocabulary Elicitation

n = 0

Is

n > 4

?

Yes

Display

view text

No

Elicit field n

Word comparison

Do

words entered

match existing

ones?

start

vocabulary

elicitation

No

Yes

n++

n: elicitation field

40

Slide 58

R.O.Weber Calgary 3 Aug 2001

41

Slide 59

R.O.Weber Calgary 3 Aug 2001

42

Slide 60

R.O.Weber Calgary 3 Aug 2001

43

Slide 61

R.O.Weber Calgary 3 Aug 2001

44

Slide 62

R.O.Weber Calgary 3 Aug 2001

45

Slide 63

R.O.Weber Calgary 3 Aug 2001

46

Slide 64

Para 5 Narr5. (U) OBSERVATION: An ODA on a low-visibility special recon mission was used to implement a NEO involving notional U.S. citizens. SOF MH-53 helicopters were used to evacuate personnel from Tinian. The original plan called for a SOC qualified MARFOR to accomplish this operation with the clandestine assistance of the SOF personnel.

Para 6 Narr6. (U) DISCUSSION: Using covert SOF personnel and helicopters to implement a NEO comprises these assets if high-visibility conventional forces are not also utilized in theoperation. An alert OPFOR commander would question exactly how the forces assisting with the NEO got on the island, if no conventional forces were inserted. He ought to be able to infer that SOF were involved, which compromises them.

Para 7 Narr7. (U) LESSON LEARNED: The CJTF must be made aware that, if he implements a plan that uses only visible SOF on a NEO without conventional forces being on the scene, this increases the risk to clandestine SOF personnel performing missions supporting his campaign plan.

Para 8 Narr8. (U) RECOMMENDED ACTION: Clandestine SOF assets on low-visibility missions generally should not be used alone to perform a NEO where they can be observed by the OPFOR.

Para 9 Narr9. (U) COMMENTS: The CJTF should use low-visibility SOF assets alone on a NEO only when the cost of leaving U.S. citizens in harm's way (possibly as hostages) exceeds the risk that compromised SOF personnel may not be able to accomplish their missions.

SOF and conventional forces

R.O.Weber Calgary 3 Aug 2001

47

Slide 65

Problems & Solutions (iii)

Lesson Elicitation Tool:

LET educates users indicating what contents to communicate

Self-explanatory elicitation tool guides users in using a pre-defined format by filling out blank fields and selecting from drop-down lists

Embedded training with examples and confirmations*

Current Collection:

Users do not know what specific content to communicate when submitting lessons

Users have to compose textual descriptions of their experiences

Lack of training and instructions for lesson submission

48

Slide 66

Next steps

  • Requirement for monitored distribution is that lessons are represented as cases.

    • Collection tool;

    • Verify methods;

    • Reasoning (?)

  • Evaluation with human subjects (simulated users in HICAP) and let human subjects decide on applying lessons.

  • Extend Monitored Distribution to other knowledge artifacts.

  • Extend Monitored Distribution to other DDS.

  • Integrate experiential knowledge with training knowledge.

  • Human issues (e.g., disclosing identity of lesson authors)

R.O.Weber Calgary 3 Aug 2001

49

Slide 67

Questions?


Copyright © 2014 SlideServe. All rights reserved | Powered By DigitalOfficePro