Metacognitive Capabilities for Advice Taking

1. Metacognitive Capabilities for Advice Taking Jihie Kim Yolanda Gil Jim Blythe Intelligent Systems Division USC/Information Sciences Institute www.isi.edu/ikcap/

2. Challenges • Systems must understand how user’s advice/guidance/request fits before accepting it. • Self-assessment of competence and confidence

3. Key Idea • Apply metacognitive skills in human learning [White and Frederickson 98,Harris 1990] Awareness of its own knowledge and the ability to understand, control and manipulate its own cognitive process

4. Approach • Explicit representation of meta-level goals to understand, control, and manipulate system’s cognitive process of KA and develop strategies for them. • Develop annotations to the dialogue history based on the goals and strategies (episodic memory) • Apply our techniques in proactive KA dialogue [Kim & Gil AIEd 2002, Gil & Kim CogSci-2001; Kim& Gil ITS-2001]

5. Meta-Level Goals Establish confidence on KB items Ensure that KB items are useful and run efficiently • Self-evaluate successful/unsuccessful procedures • procedures that are often used vs rarely used ones • procedures that are efficient/inefficient • evaluate other KB items in a similar way Be aware of the goals set (i.e. understand the purpose) Achieve goals or Ensure progress towards the goals Achieve proficiency • goals (intentions) that are often set • errors/gaps found • progress towards a goal (history) • answer changes (history) • redundant calls to a procedure

6. Meta-Level Goals (cont) Ensure that advices were useful in making the problem solving successful Ask for more advices when there are conflicts, changes in answers, redundancies, etc. Ensure that advices are consistent with existing knowledge • advices that were useful for making PS successful • advices that didn’t affect the results • frequent rejects or dislikes • contradicting advices • redundant advices Establish competence: assess own problem solving capability • kinds of problems that can be solved • kinds of problems that can’t be solved yet (based on attempts)

7. Developing Strategies for Meta-Level Goals Goals on KB items Establish confidence on KB items Ensure that KB items are useful and run efficiently • self evaluate successful/unsuccessful procedures  check the involved PS episodes, whether any of them that use the procedure succeeded Warning : find_laptop still doesn’t return any results • procedures that are often used vs rarely used ones  check how used, how many times Warning: the procedure of ‘find_midrange_systems’ hasn’t been used for a while. • procedures that are efficient/inefficient  check how used, how long it took to run Warning: calls to find_all_laptops took more than 5 min. • evaluate other KB items in a similar way

8. Developing Strategies for Meta-Level Goals (cont) Goals on Problem Solving Goals Be aware of the goals set (i.e. understand the purpose) Achieve goals or Ensure progress towards the goals Achieve proficiency • monitor goals that are often set  check the goals set in the history • goals set but not visited for a while  check the goals & PS episodes Warning: are you still interested in send_email to Susie? • errors/gaps found Warning (if error message not sent from other modules): there are still some errors … • progress towards a goal (history)  number of errors/ number of steps taken in PS episode Feedback Q: now we can perform find-laptops, but still don’t know how to do get_autorization • answer changes (history)  number of errors/ number of steps taken in PS episode Feedback: now we have IBM think pad as well as SONY VAIO as the candidates • redundant calls to a procedure Warning: in solving ‘buy laptop’ the same calls to find-laptop have been made multiple times

9. Developing Strategies for Meta-Level Goals (cont) Goals on advices taken Ensure that advices were useful in making the problem solving successful Ask for more advices when there are conflicts, changes in answers, redundancies, etc. Ensure that advices are consistent with existing knowledge • advices that were useful for making PS successful  check changes in the PS episodes Feedback: since you raised the max price, now we can find candidate laptops • advices that didn’t affect the results  check changes in the PS episodes Warning: you advice to reduce the min speed didn’t change the answer • frequent rejects or dislikes • contradicting advices •  check inconsistent results • redundant advices

10. Developing Strategies for Meta-Level Goals (cont) Goals on Overall problem solving capability Establish competence: assess own problem solving capability • kinds of problems that can be solved  monitor PSs completed and how well they are done • kinds of problems that can’t be solved yet (based on attempts)  check the answers Warning: we can’t solve find_autorization yet

11. Summary:Metacognitive Capabilities for Advice Taking Advice User Advice KB changes KB item usage Follow-up dialogue KB-1 Episodic Memory with Annotations • Meta reasoning • Self assessment of competence • Strategies for resolving conflicts • Relevance (usage) and gaps • 3/28/03 Jihie buys laptop • Initial task goals • solution proposed • advice: new cost limit • 2nd solution • . . . • 9/1/03 Jim buys laptop • . . . • 10/28/03 Jihie finds laptop KB-2 . . . KB annotations Problem Solving KB