Center for the Study of Language and Information Stanford University, Stanford, California

1. Symposium on Reasoning and Learning in Cognitive Systems Center for the Study of Language and Information Stanford University, Stanford, California March 20-21, 2004 The views contained in these slides are the author’s and do not represent official policies, either Expressed or implied, of the Defense Advanced Research Projects Agency or the DoD.

2. A number of factors encouraged us to organize this symposium: Motivation for the Symposium Reasoning and learning are both central aspects of intelligence, but the two research groups have become nearly disjoint. There exist substantial results on reasoning and learning, but many have forgotten or never learned about them. There are growing needs for integrated intelligent systems, but research focuses primarily on component technologies. DARPA now wants cognitive systems that reason and learn. We hope this meeting can help build a community of researchers that can respond to these problems and opportunities.

3. Elements of Machine Learning performance element environment knowledge learning element

4. Learning to Improve Reasoning We can state the general task of learning to improve reasoning as: Given: Initial knowledge elements for a particular domain; Given: A performance system that can compose these elements dynamically to solve problems or accomplish goals; Given: Traces of the performance system’s behavior or advice about how to solve problems in the domain; Find: New or revised knowledge elements that improve system performance on novel problems. Much of the early research on machine learning can be cast in just these terms.

5. STRIPS (1972) Anzai (1978) ACT-F (1981) Some Systems that Reason and Learn LEX (1981) SAGE (1982) UPL (1983) Soar (1984) MORRIS (1985) LEAP (1985) MacLearn (1985) Prodigy/E (1988) Eureka (1989) Bagger (1990) PRIAR (1990) Daedalus (1991) Cascade (1993) Prodigy/A (1993) SCOPE (1996)

6. Characteristics of Early Research The performance system engaged in multi-step reasoning by dynamic composition of knowledge elements. Learning methods were typically incremental and integrated with the performance system. Learning was relatively rapid and took at least some domain knowledge into account. Learning was embedded in a problem-solving architecture that made representational and performance assumptions. Research emphasized support of cognitive abilities, such as planning and reasoning, rather than perception and execution. Researchers looked to psychology and logic for ideas, rather than to statistics and operations research.

7. Some Historical Developments 1959 Creation of the General Problem Solver 1972 Development of STRIPS with MACROPs 1978 First adaptive production systems developed 1980 Carnegie symposium on learning and cognition 1981 Growth of work on learning in problem solving 1983 Active research on cognitive architectures 1986 Growth of explanation-based learning movement 1988 Recognition of the utility problem 1989 Rise of experimental method, advent of UCI repository 1991 ISLE/Stanford symposium on learning and planning 1992 Influx of ideas from pattern recognition 1993 Excitement about reinforcement learning 1995 Influx of ideas from operations research 1998 Reduced effort on learning and reasoning

8. In recent years, there have been some positive developments: Some Encouraging Signs academic courses and tutorials on learning and reasoning; AI Magazine survey of work on learning in planning domains; interest in model-based and relational reinforcement learning; broader interest in integrated cognitive architectures; DARPA workshop on rapid, embedded, and enduring learning; prospects for DARPA program in learning for cognitive systems. Taken together, these suggested the time had arrived for another meeting on reasoning and learning.

9. The meeting has some great speakers reporting on great topics, but some may wonder why there are no talks on: Some Omitted Paradigms probabilistic learning and reasoning in Bayesian networks; model-based approaches to learning from delayed reward; learning action models for use in planning and execution. Each framework can learn knowledge that supports some form of multi-step reasoning or inference. However, research in these paradigms focuses on statistical issues rather than structural ones, which we emphasize here.

10. Previous research in the area of learning and reasoning has: Some Open Research Problems focused on acquisition of relatively small knowledge bases; dealt with learning over relatively short periods of time; emphasized mental processes over action and perception; preferred logical, all-or-none frameworks over alternatives; downplayed the role of hierarchical knowledge structures; relied primarily on initial, handcrafted representations. Each of these suggests open problems that should be addressed in future projects.

11. Current learning research focuses on performance tasks that: Challenge: Learning to Improve Reasoning But many other varieties of learning instead involve: • the acquisition of modular knowledge elements that • can be composed dynamically by multi-step reasoning. involve one-step decisions for classification or regression; utilize simple reactive control for acting in the world. We should give more attention to learning such compositional knowledge. knowledge reasoning knowledge reasoning

12. Current learning research focuses on asymptotic behavior: Challenge: More Rapid Learning In contrast, humans are typically able to: • learn reasonable behavior from relatively few cases; • take advantage of knowledge to speed the learning process. We need more work on knowledge-guided learning of this variety. methods for learning classifiers from thousands of cases; methods that converge on optimal controllers in the limit. performance experience

13. Current learning research focuses on isolated induction tasks that: Challenge: Cumulative Learning In contrast, much human learning involves: • incremental acquisition of knowledge over time that • builds on knowledge acquired during earlier episodes. take no advantage of what has been learned before; provide no benefits for what is learned afterwards. We need much more research on such cumulative learning. initial knowledge extended knowledge

14. Current evaluation emphasizes static data sets for isolated tasks that: Challenge: Evaluating Embedded Learning To support the evaluation of embedded learning systems, we need: • a set of challenging environments that exercise learning and reasoning, • that include performance tasks of graded complexity and difficulty, and • that have real-world relevance but allow systematic experimental control. favor work on minor refinements of existing component algorithms; encourage mindless “bake offs” that provide little understanding. battle management in-city driving air reconnaissance

15. We are extending ICARUS, an integrated cognitive architecture that: An Advertisement: Progress on ICARUS stores long-term knowledge as hierarchical skills and concepts; encodes short-term elements as instances of long-term structures; uses numeric value functions to select skill paths for execution; modulates reactive behavior with a bias toward persistence; learns value functions for concepts and durations of skills; invokes means-ends analysis to handle unexecutable skills; learns new hierarchical skills upon resolution of impasses; Come to our poster this evening to hear more about the system.

16. End of Presentation