Advertisement
1 / 67

Intelligent Approaches to Lessons Learned Processes PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on 19-02-2012
  • Presentation posted in: General

Intelligent Approaches to Lessons Learned Processes. Rosina O Weber University of Wyoming Navy Center for Applied Research in AI, Naval Research Lab. Intelligent Approaches to Lessons Learned Processes. Collaborators: David W. Aha Hector Munoz Avila Len Breslow Nabil Sandhu. Outline. - PowerPoint PPT Presentation

Download Presentation

Intelligent Approaches to Lessons Learned Processes

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Intelligent approaches to lessons learned processes l.jpg

Intelligent Approaches to Lessons Learned Processes

Rosina O Weber

University of Wyoming

Navy Center for Applied Research in AI, Naval Research Lab


Intelligent approaches to lessons learned processes2 l.jpg

Intelligent Approaches to Lessons Learned Processes

Collaborators:

David W. Aha

Hector Munoz Avila

Len Breslow

Nabil Sandhu

R.O.Weber Calgary 3 Aug 2001


Outline l.jpg

Outline

  • Introduction

    • Context

  • Problems with lessons learned systems

  • Methodology

  • Monitored Distribution

  • Case Representation

  • Lesson Elicitation Tool

  • Next Steps

R.O.Weber Calgary 3 Aug 2001


Context l.jpg

Context

  • Knowledge management context

  • Lessons learned systems (LLS)

  • Organizations adopting LLS

  • Lessons learned definition, representation and example

  • Lessons learned process

R.O.Weber Calgary 3 Aug 2001


Knowledge management context l.jpg

Knowledge management context

  • Three types of KM initiatives*

    • knowledge repositories

    • knowledge access and transfer

    • knowledge environment

  • Knowledge repositories

    • Internet

    • industry oriented (alert systems)

    • organization oriented (lessons learned systems)

R.O.Weber Calgary 3 Aug 2001

5


Lessons learned systems l.jpg

KNOWLEDGE

ARTIFACTS

Lessons learned systems

  • Lessons learned systems are knowledge repositories of knowledge artifacts

  • Examples of knowledge artifacts are lessons, alerts, best practices, reports, video clips, etc.

R.O.Weber Calgary 3 Aug 2001

6


Lessons learned systems in organizations l.jpg

Lessons learned systems in organizations

  • Aha & Weber (Eds.)Intelligent lessons learned Systems. Papers from the AAAI 2000 Workshop(Technical Report WS-00-03) AAAI Press

  • Weber, Aha & Becerra-Fernandez (survey) Intelligent lessons learned Systems. 2001 International Journal of Expert Systems Research & Applications, Vol. 20, No. 1., 17-34.

R.O.Weber Calgary 3 Aug 2001

7


Slide8 l.jpg

government

Construction Industry Inst.

Honeywell

GM

Hewllet Packard

Bechtel Jacobs Company

Lockheed Martin E. Sys, Inc

DynMcDermott Petroleum Co.

Xerox

IBM

BestBuy

Siemens

int’l

US

European Space Agency

Italian (Alenia)

French (CNES)

Japanese (NASDA)

United Nations

Air Force

Army

Coast Guard

Joint Forces

Marine Corps

Navy

int’l

US

Department of Energy: SELLS

NASA (Ames, Goddard)

Canadian Army Lessons Learned Centre

non-government

non-military

military

8


Lessons learned definition l.jpg

Lessons learned definition…

…or organizational lessons, lessons, lessons identified

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

Definition:

A lesson learned is a knowledge or understanding gained by experience. The experience may be positive, as in a successful test or mission, or negative, as in a mishap or failure. A lesson must be significant in that it has a real or assumed impact on operations; valid in that is factually and technically correct; and applicable in that it identifies a specific design, process, or decision that reduces or eliminates the potential for failures and mishaps, or reinforces a positive result.” (Secchi et al., 1999)

R.O.Weber Calgary 3 Aug 2001

9


Lessons learned example l.jpg

Lessons learned example

applicable task

Installing custom stereo speakers.

conditions for applicability

The car is the Porsche Boxster.

lesson suggestion

Make sure you distinguish the wires leading to the speakers from the wires leading to the side airbag.

Rationale

Somebody has cut the wrong wire because they look alike and the airbag went off with explosive force. This means spending several thousand dollars to replace the airbag in addition to be a potential hazard.

From article “Learning from Mistakes” about Best Buy in Knowledge management magazine, April 2001.

R.O.Weber Calgary 3 Aug 2001

11


Lessons learned process l.jpg

Lessons learned process

R.O.Weber Calgary 3 Aug 2001

12


Lessons distribution sub process l.jpg

Lessons distribution sub process

R.O.Weber Calgary 3 Aug 2001

20


Lesson distribution methods l.jpg

Lesson distribution methods

Broadcasting

bulletins, doctrine

Passive

Standalone repository

Active casting

list servers,

information gathering tools

Pull

Push

R.O.Weber Calgary 3 Aug 2001

20


Problems with lesson distribution methods l.jpg

Problems with lesson distribution methods

  • Distribution is divorced from targeted organizational processes.

  • Users maynot know or be reminded of the repository, as they need to access a standalone tool to search for lessons.

  • Users maynot beconvinced of the potential utility of lessons.

  • Users maynot have the time and skills to retrieve and interpret relevant lessons.

  • Users maynotbe able to apply lessons successfully.

R.O.Weber Calgary 3 Aug 2001

21


Here is a gap l.jpg

Organization’s

members

Repository of lessons learned

Here is a gap

Organizational processes

R.O.Weber Calgary 3 Aug 2001


How to bridge this gap l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide17 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide18 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide19 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide20 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide21 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide22 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide23 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Slide24 l.jpg

Organization’s

members

Repository of lessons learned

How to bridge this gap?

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Monitored distribution l.jpg

Organization’s

members

Repository of lessons learned

Monitored distribution

Organizational processes

R.O.Weber Calgary 3 Aug 2001


Monitored distribution26 l.jpg

Organization’s

members

Repository of lessons learned

Monitored distribution

Lesson repository is integrated with targeted processes

…and lessons aredistributed when and where they are needed.

Organizational processes

R.O.Weber Calgary 3 Aug 2001

22


Problems with lessons learned systems l.jpg

Problems with lessons learned systems

  • Technological

  • Human

  • Managerial

R.O.Weber Calgary 3 Aug 2001

13


Technological issues l.jpg

Technological issues

Standalone distribution outside context of reuse

Lessons disseminated in context of reuse

Low precision and recall in text databases

Case retrieval for lesson dissemination

  • cases indexed by applicability

    Convert lessons into cases/Collect cases

    Requirement: machine recognizable format

    Textual representation of lessons

    Case strutcture for lessons

R.O.Weber Calgary 3 Aug 2001

14


Human issues l.jpg

Human issues

Lesson authors

Lesson validators

Lesson (re)users

  • Lack of training/instructions: content and format

  • Hard to validate textual descriptions

  • Have to access the repository in another context

  • Have to accept the potential benefit of lessons

  • Have the skills to search for lessons

  • Have to interpret textual lessons

R.O.Weber Calgary 3 Aug 2001

15


Slide30 l.jpg

Managerial issues

  • Determine, communicate and enforce standards for lesson collection and representation

  • Define structured format

  • Embed knowledge in targeted processes

  • Monitor knowledge transfer

  • Oversee knowledge reuse

Knowledge collection

Knowledge validation

Knowledge reuse

R.O.Weber Calgary 3 Aug 2001

16


Technological human managerial issues l.jpg

Monitored distribution

Case retrieval

Lessons as structured cases

Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format

Technological, Human, Managerial issues

Standalone repository

Retrieval method

Textual format of lessons

Collection method

embed instructions

for lesson submission

R.O.Weber Calgary 3 Aug 2001

17


Methodology l.jpg

Methodology

Prescriptive KM infra structure for frameworks

human users

  • Monitored distribution

  • Case representation

  • Lesson elicitation tool that embeds instructions for lesson submission and converts them into the structured format

LET

organizational processes

lesson repository for

monitored distribution

Case base

case

of lessons

base

R.O.Weber Calgary 3 Aug 2001

18


Frameworks l.jpg

Frameworks

  • Monitored Distribution

  • Case represebtation

  • Lesson Elicitation Tool

R.O.Weber Calgary 3 Aug 2001

19


Monitored distribution34 l.jpg

Monitored Distribution


Problems solutions l.jpg

Problems & Solutions

Lessons are integrated to targeted organizational processes.

Users don’t need to be reminded of the repository because they don’t need to access a standalone tool.

No additional time or skills are required.

Case retrieval of disambiguated knowledge increase recall and precision

Whenever possible, an ‘apply’ button allows the lesson to be automatically executable.

Distribution is divorced from targeted organizational processes.

Users maynot be reminded of the repository, as they need to access a standalone tool to search for lessons.

Users maynot have the time and skills to retrieve relevant lessons.

Text databases have low levels of precision and recall

Users maynotbe able to apply lessons successfully.

20


Monitored distribution characteristics l.jpg

Monitored distribution characteristics

  • Distribution is tightly integrated to the targeted processes and distribute lessons when and wherethey are needed.

  • Represent lessons as form-like cases.

  • Distribute lessons using case-retrieval/ and retrieve lessons based on similarity.

  • Additional benefits are:

  • Case representation facilitates interpretation.

  • Users access the lesson rationale to evaluate its potential utility.

  • Cases are retrieved in the context of similar experiences.

R.O.Weber Calgary 3 Aug 2001

21


Slide37 l.jpg

Noncombatant Evacuation Operations:

Military operations to evacuate noncombatants whose lives are in danger and rescue them to a safe haven

R.O.Weber Calgary 3 Aug 2001


Slide38 l.jpg

Assembly Point

Campaign headquarters

Intermediate Staging Base

.

safe haven

NEO site


Example in hicap l.jpg

Example in HICAP

  • HICAP is a plan authoring tool suite http://www.aic.nrl.navy.mil/hicap

  • Muñoz-Avila et al., 1999

  • Users interact with HICAP by refining an HTN (hierarchical task network) through decompositions

R.O.Weber Calgary 3 Aug 2001

24


Slide40 l.jpg

safe haven

NEO site

R.O.Weber Calgary 3 Aug 2001


Selecting the suggested case l.jpg

Selecting the Suggested Case…

R.O.Weber Calgary 3 Aug 2001


Expanding yields l.jpg

Expanding yields…

R.O.Weber Calgary 3 Aug 2001


And the user is notified of a lesson l.jpg

And the user is notified of a lesson

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing them.

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing them.

RATIONALE:

TYPE: advice

Clandestine SOF should not be used alone

WHY: The enemy might be able to infer that SOF are involved, exposing

WHY: The enemy might be able to infer that SOF are involved, exposing

R.O.Weber Calgary 3 Aug 2001


After applying the lesson l.jpg

After applying the lesson

R.O.Weber Calgary 3 Aug 2001


Evaluation l.jpg

Evaluation

  • Hypothesis

    • Using lessons will improve plan quality

  • Methodology

    • Simulated HICAP users generated NEO plans with and without lessons

    • NEO executor implemented plans

      • Plan total duration

      • Plan duration before medical assistance

      • Casualties: evacuees, FF, enemies

R.O.Weber Calgary 3 Aug 2001

30


Plan evaluator l.jpg

Plan evaluator

  • non deterministic (100 plans 10 times each)

  • 30 variables: 12 random

  • length of plans 18 steps

  • size of planning space 3,000,000

  • 13 lessons

  • Actions and their influences

R.O.Weber Calgary 3 Aug 2001

31


Plan evaluator actions l.jpg

Plan evaluator: actions

  • Plans where evacuees were transported by land modes have an increased chance of being attacked by enemies.

  • Plans that combined weather with too strong wings have a small chance of helicopter crash.

  • When an attack or crash happens it increases the number of casualties among evacuees and FF (in proportion to # of evacuees).

R.O.Weber Calgary 3 Aug 2001

32


Plan evaluator lessons l.jpg

Plan evaluator: lessons

Conditions for applicability:

There are representatives of different branches assigned to participate.

Lesson suggestion:

Assign representatives of all forces to plan.

Rationale:Lack of representatives prevent good communication causing delays and miscommunication.

  • Conditions for applicability:

  • There are hundreds or more evacuees as to justify a security effort.

  • Lesson suggestion:

  • Assign EOD personnel.

  • Rationale:An evacuee once asked what to do with their weapons.

R.O.Weber Calgary 3 Aug 2001

33


Results l.jpg

Results

*The resulting values are averages

no lessons

with lessons

reduction

NEO plan

total duration*

32h48

18 %

39h50

duration until

medical assistance*

24h13

18 %

29h37

casualties

among evacuees

24 %

11.48

8.69

casualties among

friendly forces

6.57

30 %

9.41

casualties

among enemies

-2 %

3.08

3.14

R.O.Weber Calgary 3 Aug 2001

34


Case representation l.jpg

Case representation


Cbr cycle and knowledge processes l.jpg

CBR Cycle and Knowledge Processes

distribute

store

New

knowledge

collection

verify


Case representation52 l.jpg

Case Representation

case problem indexing elements

case solutionreuse elements

Distributing process knowledge in the context of the targeted process.

R.O.Weber Calgary 3 Aug 2001

36


Lessons learned representation l.jpg

Lessons learned representation

  • indexing elements:

    • applicable task

    • conditions for applicability

  • reuse elements:

    • lesson suggestion

    • rationale

  • indexing elements:

    • applicable task

    • conditions for applicability

  • reuse elements:

  • indexing elements:

  • reuse elements:

R.O.Weber Calgary 3 Aug 2001

37


Problems solutions54 l.jpg

Problems & Solutions

If more than one lesson happens to be retrieved, the representation allows the user to assess its relevance immediately

Pre-defined content and format limit the content to what is applicable, facilitating correct use

The conversion into cases forces disambiguation facilitating interpretation

Structured representation can potentially allow automatic verifications, without adding unnecessary content

Users maynotbe able to correctly interpret retrieved lessons because they may be long and not well written

Free text collection allows unlimited content and have the potential to cause misuse of lessons

Users may not be able to interpret long and ambiguous texts

Textual representations complicate validation, sometimes validation results in more text added

38


Lesson elicitation tool l.jpg

Lesson Elicitation Tool


Elicitation tool l.jpg

Elicitation Tool

  • LET is supported by generic and domain-specific lexicons of expressions and verbs (do not store unless relevant)

  • Confirmations search for clues indicating user’s need for help

  • A domain lexicon supports disambiguation at run-time Uses a subset of NL based on the case representation by using a template-based elicitation with pre-defined grammar structures thus avoiding NL parsing

  • LET supports conversation to acquire new concepts for the ontology

R.O.Weber Calgary 3 Aug 2001

39


Slide57 l.jpg

1.2 Fields Elicitation

1.2.1 APPLICABLE ACTION Elicitation

1.2.2 SUGGESTION Elicitation

1.2.3 CONDITIONS Elicitation

1.2.4 ORIGINATING EVENT Elicitation

1.2.5 View text (edit)

1.2.a Word comparison

1.2.b Vocabulary Elicitation

n = 0

Is

n > 4

?

Yes

Display

view text

No

Elicit field n

Word comparison

Do

words entered

match existing

ones?

start

vocabulary

elicitation

No

Yes

n++

n: elicitation field

40


Slide58 l.jpg

R.O.Weber Calgary 3 Aug 2001

41


Slide59 l.jpg

R.O.Weber Calgary 3 Aug 2001

42


Slide60 l.jpg

R.O.Weber Calgary 3 Aug 2001

43


Slide61 l.jpg

R.O.Weber Calgary 3 Aug 2001

44


Slide62 l.jpg

R.O.Weber Calgary 3 Aug 2001

45


Slide63 l.jpg

R.O.Weber Calgary 3 Aug 2001

46


Sof and conventional forces l.jpg

Para 5 Narr5. (U) OBSERVATION: An ODA on a low-visibility special recon mission was used to implement a NEO involving notional U.S. citizens. SOF MH-53 helicopters were used to evacuate personnel from Tinian. The original plan called for a SOC qualified MARFOR to accomplish this operation with the clandestine assistance of the SOF personnel.

Para 6 Narr6. (U) DISCUSSION: Using covert SOF personnel and helicopters to implement a NEO comprises these assets if high-visibility conventional forces are not also utilized in theoperation. An alert OPFOR commander would question exactly how the forces assisting with the NEO got on the island, if no conventional forces were inserted. He ought to be able to infer that SOF were involved, which compromises them.

Para 7 Narr7. (U) LESSON LEARNED: The CJTF must be made aware that, if he implements a plan that uses only visible SOF on a NEO without conventional forces being on the scene, this increases the risk to clandestine SOF personnel performing missions supporting his campaign plan.

Para 8 Narr8. (U) RECOMMENDED ACTION: Clandestine SOF assets on low-visibility missions generally should not be used alone to perform a NEO where they can be observed by the OPFOR.

Para 9 Narr9. (U) COMMENTS: The CJTF should use low-visibility SOF assets alone on a NEO only when the cost of leaving U.S. citizens in harm's way (possibly as hostages) exceeds the risk that compromised SOF personnel may not be able to accomplish their missions.

SOF and conventional forces

R.O.Weber Calgary 3 Aug 2001

47


Problems solutions iii l.jpg

Problems & Solutions (iii)

Lesson Elicitation Tool:

LET educates users indicating what contents to communicate

Self-explanatory elicitation tool guides users in using a pre-defined format by filling out blank fields and selecting from drop-down lists

Embedded training with examples and confirmations*

Current Collection:

Users do not know what specific content to communicate when submitting lessons

Users have to compose textual descriptions of their experiences

Lack of training and instructions for lesson submission

48


Next steps l.jpg

Next steps

  • Requirement for monitored distribution is that lessons are represented as cases.

    • Collection tool;

    • Verify methods;

    • Reasoning (?)

  • Evaluation with human subjects (simulated users in HICAP) and let human subjects decide on applying lessons.

  • Extend Monitored Distribution to other knowledge artifacts.

  • Extend Monitored Distribution to other DDS.

  • Integrate experiential knowledge with training knowledge.

  • Human issues (e.g., disclosing identity of lesson authors)

R.O.Weber Calgary 3 Aug 2001

49


Slide67 l.jpg

Questions?