Cognitive Computer Vision

1. Cognitive Computer Vision Kingsley Sage khs20@sussex.ac.uk and Hilary Buxton hilaryb@sussex.ac.uk Prepared under ECVision Specific Action 8-3 http://www.ecvision.org

2. Course outline • What is Cognitive Computer Vision (CCV) ? • Generative models • Graphical models • Techniques for modelling cognitive aspects of CCV • Bayesian inference • Markov Models • Research issues • Coursework and case studies

3. So what is CCV ?

4. So what is CCV ? • In this course, we focus on using of ideas from cognitive science and psychology to do CCV • To show how we can build effective CCV systems that are more robust and more capable of solving non-trivial problems than those that do not embrace these ideas • Use statistical inference and machine learning as our tools for modelling cognitively inspired processes • We are not claiming “hard AI” in this course

5. Key Cognitive Elements • Objects, events, activities and behaviours • “What is it that we are observing?” • Attention and control • “How is it that we observe?”

6. Key Cognitive Elements • Visual learning and memory • Representation of objects and their behaviour • Recognition • Categorisation • These are “what” problems • Visual control and attention • Perception for tasks using models of expectation • Goals, task context • Resources, embodiment • These are “how” problems • Cognition • From perception to action

7. Key Cognitive Elements • Visual learning and memory - examples • Learning about objects and how their appearance can change • Recognising activities by the interactions between objects • Extracting invariant models from training data

8. Learning and “recognising” objects (Murase and Nayar, 1996)

9. Learn and recognise activities Coupled Hidden Markov Models (CHMM) techniques (Oliver, Rosario & Pentland, 1999) Activities with interactions via coupled states in a HMM

10. Learning invariant models Means for 3 clusters Variances for 3 clusters

11. Key Cognitive Elements • Visual control and attention • A framework for attentional control • Inferring likely behaviour using Bayes nets • Deictic markers • Attentional selection of objects

12. Task Based Control CONTROL POLICY (WITH STATE MEMORY) FEATURE COMBINATION …… dN d1 d2 Image Data Driven A Framework For Task Based Visual Control Scene Interpretation

13. IGP orient size lo2 ls1 ls2 lo1 BBN Inference of likely vehicle tracks Gong and Buxton, 1993 Fixed camera gives direct set of dependencies Image Grid Position BBN has size/orient hidden nodes Leaf nodes ls1/2, lo1/2 observables

14. Deictic Markers in inference of behaviour Howarth and Buxton,1996 Left: attention for overtake (overtaken & overtaking vehicle) Right: attention for giveway (stopped & blocker vehicle plus ground-plane conflict zone)

15. Attentional selection using eye gaze

16. Attentional selection using predicted trajectory data

17. Attentional selection using predicted trajectory data

18. Attentional selection using predicted Space of Interest

19. Summary • Cognitive Computer Vision is a multi-disciplinary area of research • Here we use statistical inference and learning for robust models • Task based attentional control is key to prediction and cognitive systems design • Useful reference: “Visual surveillance in a dynamic and uncertain world” Buxton, H and Gong, S, Artificial Intelligence 78, pp 431-459, 1995

20. Next time … • Generative models • What are they? • Why are they so important to Cognitive Vision?