WP5 Cognitive Integration and Bootstrapping

WP5 Cognitive Integration and Bootstrapping Angelo Cangelosi,Alessandra Sciutti, Metta, Sagerer, Nolfi, Nehaniv, Fischer ,Tani, Belpaeme, Fadiga, Nori, Rohlfing, Wrede, Tuci, Tikhanoff, Schillingmann, Lohan, Mirolli, Saunders, Dautenhahn, Zeschel, Marocco, Peniak, Foerster, Massera, Sato, Lyon, Pitsch, Gijsberts, Beck, et al.

Overview WP5 Overview: The integration WP? Task 5.1 (Roadmap) Road Map preparation Results and dissemination Task 5.3 (HRI and neuroscience) Planned experiments Main result(s) Task 5.2 (language as cognitive tool) Planned experiments Integration Outlook ITALK Year 1 ReviewDüsseldorf, 1 July 2009

The Integration WP? • Different levels of integration of work • Intra-WP: Each WP includes integration of cognitive capabilities • E.g. WP1 action withlanguage (T1.4  T4.3) • E.g. WP3 social withlanguage (T3.1) • Inter-WP: WP5 for additional aspects of cognitive integration (bootstrapping) • Roadmap vision (T5.1) • Extra-linguistic roles of language (T5.2) • From neuroscience to HRI (T5.3, T5.4) • Work (eg. results, software) integration • Standardisation process (iCub/ITALK repository)

WP5 Objectives • WP Objectives (Year 1) • to identify the underlying methodological, theoretical and technological principlesfor research on the integration of linguistic, motor and cognitive capabilities in robots; • to uncover, from both a theoretical and an application point of view, the relationships between the different levels of integration and transfer of action and language knowledge through the involvement of the mirror-neuron system and the role of motor affordances in human-robot communication experiments • the understand the role of language as a cognitive tool and its role in the bootstrapping of cognitive capabilities such as attention and gaze sharing, syntax processing, and object representation through motor affordances ROADMAP Gaze experiments Language as cognitive tool

ROADMAP: Process • Meeting 1: Planning of process • Meeting 2: Rome Workshop • Drafting • 5 Thematic groups • Email co-writing • Dissemination • Website • Submission to IEEE TAMD journal

ROADMAP: Intl. Workshop • Participants • All ITALK principal scientists • ISAB/Advisors: • I. Pepperberg (Harvard USA) • M. Bowermann (Max Planck) • A. Glenberg (Arizona State) • R. Kubat (MIT USA, Roy’s lab) • K. Plunkett (Oxford UK) • S. Harnad (UQAM & Southampton University.) • R. Grush (UCSD USA) • Related EU projects: • L. Steels (ALEAR), T. Ziemke (ICEA), A. Borghi (ROSSI), P. De Ruiter (JAST), P. Dominey (CHRIS)

ROADMAP: Intl. Workshop

ROADMAP: Post-Workshop • Thematic Groups • Introduction/Vision/Milestones: led by A. Cangelosi (PLYM) with contribution from T. Belpaeme and all other staff • Action Learning Challenges: led by G. Metta and L. Fadiga (IIT) with contribution from PLYM staff • Category Learning Challenges: led by S. Nolfi (CNR) with contribution from PLYM and RIKEN staff • Social Learning Challenges: led by G. Sagerer (BIEL) with contribution from UH and USD staff • Language Learning Challenges: led by K. Fischer (USD) with contribution from BIEL, PLYM and RIKEN staff

ROADMAP Results:Key Challenges • Understanding how agents learn and represent compositional and hierarchical actions; • Understanding how agents acquire and represent compositional lexicons and linguistic knowledge; • Understanding the dynamics of social interaction and social learning, • Understanding of how compositional action and language representations are integrated to bootstrap cognition.

ROADMAP Results: Milestones

ROADMAP: Publication • Roadmap white paper (website) • Publication (submitted) • Workshop proceedings (CDROM and on-line)

Task 5.3: goals • Human-robot experiments and mirror-neuron system: Gaze behaviour during observation of actions performed by a humanoid artefact • Human-human interaction: mo 6-24 • Human-robot interaction: mo 24-30 ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Proactive gaze behaviour and mirror neurons • Gaze behaviour is proactive (Flanagan and Johansson 2003)  fundamental in interactions (also human – robot). • Which are the characteristics of the observed action that evoke the pro-active gaze behaviour? Eye movements are used to reflect the activity of a motor resonance involving a mirror-neuron system. • Proactive gaze behavior as a function of: • Observed action kinematics • Cognitive cues ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Experiments Effects of the observation of liftingobjects of different weight on: • Mirror neurons involvement • Previous experiment • Proactive gaze behaviour • New ongoing pilot experiment ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Object lifting task Optotrak cameras Optotrak markers Eyelink system ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Stimuli Object appearance Demonstrator’s kinematics Visible heavy vs. Visible light Hidden heavy vs. Hidden light Demonstrator’s kinematics only Labeled heavy vs. Labeled light (both heavy) Object appearance only: cognitive cues ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Mirror neurons involvement Normalized MEP area (Z-score) EMG : FDI TMS : 120% of MT 50 ms after lifting Protocol: 8 subjects, 10 trials,Conditions randomized ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Hand – Gaze Measures ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Preliminary results Light Placing Lifting HAND Kinematics Heavy HAND & GAZE Observer Demonstrator ITALK Year 1 ReviewDüsseldorf, 1 July 2009

Current and future steps • Currently: setup ready, starting data collection. • Next steps: • Modulating predictability of demonstrator’s motion (e.g. alternating different arrival points) • Robot as demonstrator • Modulation of kinematics • Modulation of interaction ITALK Year 1 ReviewDüsseldorf, 1 July 2009

T5.2: Language as a cognitive tool and its role in enhancing cognition (e.g. inner speech) The aim of this task is that to verify, through robotic experiments, whether: • the co-development of behavioral and linguistic skills represent a crucial prerequisite for the development of complex cognitive skills • linguistically mediated social interactions cause a radical transformation of elementary cognitive abilities, as proposed by Vygotsky (1962, 1978) • Talking-to-oneself (inner speech) as a cognitive aid tool (Vygotsky 1978)

Objects manipulation scenario • A robot located in front of few objects will be trained (through a trial and error process) for displaying: • few behavioural skills (e.g. “touching the blue object”, “indicating the red object”, or “moving the green object”) • Associated linguistic skills that will consists in “verbally describing the behaviour that the robot is currently exhibiting” and/or “listening to a verbal command and executing the corresponding behaviour”

Shape recognition/drawing scenario • A robot located in front of a white board will be trained (through a trial and error process)for: • Drawing a shapes (e.g. simple geometrical shapes such us a square or a circle or more elaborate shapes) by moving a pen after having observed the experimenter drawing the same shape • Verbally describing the shape being drawn by the experimenter or by the robot itself

Research Objectives • Through these experiments we will investigate: • Whether the possibility to be exposed to linguistic inputs might facilitate the development of the required behavioural and cognitive skills. • Whether the need to produce linguistic outputs can facilitate the development of the required behavioural and cognitive skills • Whether the possibility of the agent to access own linguistic outputs (inner speech) can represent a prerequisite for the development of certain behavioral and cognitive skills • How the internal categories developed by the agents are co-shaped by the development of behavioral and linguistic skills

WP5 Outlook: Integration for Cognitive Bootstrapping • Additional dimensions in cognitive integration • E.g. Neuroscience and mirror neuron system (Tasks 5.3-5.4) Mirror neuron involvement on gaze in humans   Mirror neuron involvement on gaze in HRI   Mirror neuron involvement on linguistic complexity in HRI • E.g. Self-talk (Tasks 5.2) Language for communication   Language for taking-to-oneself  • Language as cognitive aid tool  • language/cognition relativism

WP5 Cognitive Integration and Bootstrapping