Some Information

1. Some Information Parking is free in Stanford lots over the weekend. There is no registration fee thanks to ONR and NSF. Participation in the meeting is by invitation only. We will provide lunch on both days to registered participants. We will provide dinner on Saturday to registered participants. Restrooms are in the hall to the right of the Cordura lobby. Some Requests Speakers: Please limit your talks to the allotted 35 minutes. Audience: Please ask speakers only one question at a time. Please recycle bottles, drinks, and paper in labeled receptacles. Please do not smoke in or near the CSLI buildings or events.

2. Reasoning and Learning in Cognitive Systems Pat Langley Center for the Study of Language and Information Stanford University, Stanford, California http://cll.stanford.edu/~langley langley@csli.stanford.edu The views contained in these slides are the author’s and do not represent official policies, either Expressed or implied, of the Defense Advanced Research Projects Agency or the DoD.

3. A number of factors encouraged us to organize this symposium: Motivation for the Symposium Reasoning and learning are both central aspects of intelligence, but the two research groups have become nearly disjoint. There exist substantial results on reasoning and learning, but many have forgotten or never learned about them. There are growing needs for integrated intelligent systems, but research focuses primarily on component technologies. DARPA now wants cognitive systems that reason and learn. We hope this meeting can help build a community of researchers that can respond to these problems and opportunities.

4. Elements of Machine Learning performance element environment knowledge learning element

5. Learning to Improve Reasoning We can state the general task of learning to improve reasoning as: Given: Initial knowledge elements for a particular domain; Given: A performance system that can compose these elements dynamically to solve problems or accomplish goals; Given: Traces of the performance system’s behavior or advice about how to solve problems in the domain; Find: New or revised knowledge elements that improve system performance on novel problems. Much of the early research on machine learning can be cast in just these terms.

6. STRIPS (1972) Anzai (1978) ACT-F (1981) Some Systems that Reason and Learn LEX (1981) SAGE (1982) UPL (1983) Soar (1984) MORRIS (1985) LEAP (1985) MacLearn (1985) Prodigy/E (1988) Eureka (1989) Bagger (1990) PRIAR (1990) Daedalus (1991) Cascade (1993) Prodigy/A (1993) SCOPE (1996)

7. Characteristics of Early Research The performance system engaged in multi-step reasoning by dynamic composition of knowledge elements. Learning methods were typically incremental and integrated with the performance system. Learning was relatively rapid and took at least some domain knowledge into account. Learning was embedded in a problem-solving architecture that made representational and performance assumptions. Research emphasized support of cognitive abilities, such as planning and reasoning, rather than perception and execution. Researchers looked to psychology and logic for ideas, rather than to statistics and operations research.

8. Some Historical Developments 1959 Creation of the General Problem Solver Development of STRIPS with MACROPs

9. Some Historical Developments 1959 Creation of the General Problem Solver 1972 Development of STRIPS with MACROPs 1978 First adaptive production systems developed 1980 Carnegie symposium on learning and cognition 1981 Growth of work on learning in problem solving Active research on cognitive architectures

10. Some Historical Developments 1959 Creation of the General Problem Solver 1972 Development of STRIPS with MACROPs 1978 First adaptive production systems developed 1980 Carnegie symposium on learning and cognition 1981 Growth of work on learning in problem solving 1983 Active research on cognitive architectures 1986 Growth of explanation-based learning movement 1988 Recognition of the utility problem 1989 Rise of experimental method, advent of UCI repository 1991 ISLE/Stanford symposium on learning and planning

11. Some Historical Developments 1959 Creation of the General Problem Solver 1972 Development of STRIPS with MACROPs 1978 First adaptive production systems developed 1980 Carnegie symposium on learning and cognition 1981 Growth of work on learning in problem solving 1983 Active research on cognitive architectures 1986 Growth of explanation-based learning movement 1988 Recognition of the utility problem 1989 Rise of experimental method, advent of UCI repository 1991 ISLE/Stanford symposium on learning and planning 1992 Influx of ideas from pattern recognition 1993 Excitement about reinforcement learning 1995 Influx of ideas from operations research 1998 Reduced effort on learning and reasoning

12. In recent years, there have been some positive developments: Some Encouraging Signs academic courses and tutorials on learning and reasoning; AI Magazine survey of work on learning in planning domains; interest in model-based and relational reinforcement learning; broader interest in integrated cognitive architectures; DARPA workshop on rapid, embedded, and enduring learning; prospects for DARPA program in learning for cognitive systems. Taken together, these suggested the time had arrived for another meeting on reasoning and learning.

13. The meeting has some great speakers reporting on great topics, but some may wonder why there are no talks on: Some Omitted Paradigms Probabilistic learning and reasoning in Bayesian networks; Model-based approaches to learning from delayed reward; Learning action models for use in planning and execution. Each framework can learn knowledge that supports some form of multi-step reasoning or inference. However, research in these paradigms focuses on statistical issues rather than structural ones, which we emphasize here.

14. Previous research in the area of learning and reasoning has: Some Open Research Problems focused on acquisition of relatively small knowledge bases; dealt with learning over relatively short periods of time; emphasized mental processes over action and perception; preferred logical, all-or-none frameworks over alternatives; downplayed the role of hierarchical knowledge structures; relied primarily on initial, handcrafted representations. Each of these suggests open problems that should be addressed in future projects.

15. Current learning research focuses on performance tasks that: Challenge: Learning to Improve Reasoning But many other varieties of learning instead involve: • the acquisition of modular knowledge elements that • can be composed dynamically by multi-step reasoning. involve one-step decisions for classification or regression; utilize simple reactive control for acting in the world. We should give more attention to learning such compositional knowledge. knowledge reasoning knowledge reasoning

16. Current learning research focuses on asymptotic behavior: Challenge: More Rapid Learning In contrast, humans are typically able to: • learn reasonable behavior from relatively few cases; • take advantage of knowledge to speed the learning process. We need more work on knowledge-guided learning of this variety. methods for learning classifiers from thousands of cases; methods that converge on optimal controllers in the limit. performance experience

17. Current learning research focuses on isolated induction tasks that: Challenge: Cumulative Learning In contrast, much human learning involves: • incremental acquisition of knowledge over time that • builds on knowledge acquired during earlier episodes. take no advantage of what has been learned before; provide no benefits for what is learned afterwards. We need much more research on such cumulative learning. initial knowledge extended knowledge

18. Current evaluation emphasizes static data sets for isolated tasks that: Challenge: Evaluating Embedded Learning To support the evaluation of embedded learning systems, we need: • a set of challenging environments that exercise learning and reasoning, • that include performance tasks of graded complexity and difficulty, and • that have real-world relevance but allow systematic experimental control. favor work on minor refinements of existing component algorithms; encourage mindless “bake offs” that provide little understanding. battle management in-city driving air reconnaissance

19. We are extending ICARUS, an integrated cognitive architecture that: An Advertisement: Progress on ICARUS stores long-term knowledge as hierarchical skills and concepts; encodes short-term elements as instances of long-term structures; uses numeric value functions to select skill paths for execution; modulates reactive behavior with a bias toward persistence; learns value functions for concepts and durations of skills; invokes means-ends analysis to handle unexecutable skills; learns new hierarchical skills upon resolution of impasses; Come to our poster this evening to hear more about the system.

20. Some Information Parking is free in Stanford lots over the weekend. There is no registration fee thanks to ONR and NSF. Participation in the meeting is by invitation only. We will provide lunch on both days to registered participants. We will provide dinner on Saturday to registered participants. Restrooms are in the hall to the right of the Cordura lobby. Some Requests Speakers: Please limit your talks to the allotted 35 minutes. Audience: Please ask speakers only one question at a time. Please recycle bottles, drinks, and paper in labeled receptacles. Please do not smoke in or near the CSLI buildings or events.

21. End of Presentation

22. The field of machine learning has many success stories, but: Expanding our Computational Horizons Instead, we need a new vision for machine learning technology that: • supports the construction of general intelligent systems; • aspires to the same learning abilities as appear in humans. these successes are prime examples of niche AI, which develops techniques that are increasingly powerful but that apply to an ever narrower classes of problems. This would produce a broader research agenda that would take the field into unexplored regions. niche AI power cognitive systems generality

23. Current learning research emphasizes tasks like classification and reactive control, whereas humans learn: Challenge 1: More Varied Learning • grammars for understanding natural language; • heuristics for reasoning and problem solving; • scripts and procedures for routine behavior; • cognitive maps for localization and navigation; • models that explain the behavior of artifacts. We need more work on learning such varied knowledge structures. human learning abilities current focus of machine learning