Motivated learning based on goal creation
Download
1 / 34

Motivated Learning based on Goal Creation - PowerPoint PPT Presentation


  • 57 Views
  • Uploaded on

Motivated Learning based on Goal Creation. Janusz Starzyk School of Electrical Engineering and Computer Science, Ohio University, USA www.ent.ohiou.edu/~starzyk. Istituto Dalle Molle di Studi sull'Intelligenza Artificiale , 4 December 2009. Outline. Embodied Intelligence (EI)

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Motivated Learning based on Goal Creation' - tad-higgins


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Motivated learning based on goal creation
Motivated Learning based on Goal Creation

Janusz Starzyk

School of Electrical Engineering and Computer Science, Ohio University, USA

www.ent.ohiou.edu/~starzyk

Istituto Dalle Molle di Studi sull'Intelligenza Artificiale, 4 December 2009.


Outline

  • Embodied Intelligence (EI)

  • Embodiment of Mind

  • How to Motivate a Machine

  • Goal Creation Hierarchy

  • GCS Experiment

  • Motivated Learning


Design principles of intelligent systems
Design principles of intelligent systems

from Rolf Pfeifer “Understanding of Intelligence”, 1999

  • Interaction with complex environment

  • cheap design

  • ecological balance

  • redundancy principle

  • parallel, loosely coupled processes

  • asynchronous

  • sensory-motor

    coordination

  • value principle

Agent

Drawing by Ciarán O’Leary- Dublin Institute of Technology


Embodied intelligence
Embodied Intelligence

  • Definition

  • Embodied Intelligence (EI) is a mechanism that learns how to survive in a hostile environment

  • Mechanism: biological, mechanical or virtual agent

    with embodied sensors and actuators

  • EI acts on environment and perceives its actions

  • Environment hostility is persistent and stimulates EI to act

  • Hostility: direct aggression, pain, scarce resources, etc

  • EI learns so it must have associative self-organizing memory

  • Knowledge is acquired by EI


Embodiment of a mind
Embodiment of a Mind

  • Embodiment is a part of environment under control of the mind

  • It contains intelligence core and sensory motor interfaces to interact with environment

  • It is necessary for development of intelligence

  • It is not necessarily constant


Embodiment of Mind

  • Changes in embodiment modify brain’s self-determination

  • Brain learns its own body’s dynamics

  • Self-awareness is a result of identification with own embodiment

  • Embodiment can be extended by using tools and machines

  • Successful operation is a function of correct perception of environment and own embodiment


How to motivate a machine
How to Motivate a Machine ?

A fundamental question is what motivates an agent to do anything, and in particular, to enhance its own complexity?

What drives an agent to explore the environment and learn ways to effectively interact with it?


How to motivate a machine1
How to Motivate a Machine ?

  • Pfeifer claims that an agent’s motivation should emerge from the developmentalprocess.

    • He called this the “motivated complexity” principle.

    • Chicken and egg problem? An agent must have a motivation to develop while his motivation comes from development?

  • Steels suggested equipping an agent with self-motivation.

    • “Flow” experienced when people perform their expert activity well would motivate to accomplish even more complex tasks.

    • But what is the mechanism of “flow”?

  • Oudeyer proposed an intrinsic motivation system.

    • Motivation comes from a desire to minimize the prediction error.

    • Similar to “artificial curiosity” presented by Schmidhuber.


How to motivate a machine2
How to Motivate a Machine ?

  • Although artificial curiosity helps to explore the environment, it leads to learning without a specific purpose.

    • It may be compared to exploration in reinforcement learning.

  • Exploration is needed in order to learn and to model the environment.

    • But is exploration the only motivation we need to develop EI?

    • Can we find a more efficient mechanism for learning?

  • I suggest a simpler mechanism to motivate a machine.


How to motivate a machine3
How to Motivate a Machine ?

  • I suggest that it is the hostility of the environment, in the definition of EI that is the most effective motivational factor.

    • It is the pain we receive that moves us.

    • It is our intelligence determined to reduce this pain that motivates us to act, learn, and develop.

  • Both are needed - hostility of the environment and intelligence that learns how to reduce the pain.

    • Thus pain is good.

    • Without pain we would not be motivated to develop.

Fig. englishteachermexico.wordpress.com/


Motivated learning
Motivated Learning

  • I suggest a goal-driven mechanism to motivate a machine to act, learn, and develop.

    • A simple pain based goal creation system.

    • It uses externally defined pain signals that are associated with primitive pains.

    • Machine is rewarded for minimizing the primitive pain signals.

  • Definition: Motivated learning (ML) is learning based on the self-organizing system of goal creation in embodied agent.

    • Machine creates abstract goals based on the primitive pain signals.

    • It receives internal rewards for satisfying its goals (both primitive and abstract).

    • ML applies to EI working in a hostile environment.


Pain center and goal creation
Pain-center and Goal Creation

  • Simple Mechanism

  • Creates hierarchy

  • of values

  • Leads to formulation

  • of complex goals

  • Reinforcement

    • Pain increase

    • Pain decrease

  • Forces exploration


Abstract goal creation for ml

Sensory pathway

Motor pathway

(perception, sense)

(action, reaction)

refrigerator

Open

Level II

-

+

food

becomes a

Abstract pain

sensory input to

(Delayed memory of pain)

abstract pain center

Food

Eat

Level I

-

+

Dual pain

Pain

Primitive

Level

Stomach

Association

Inhibition

Reinforcement

Connection

Planning

Expectation

Abstract Goal Creation for ML

  • The goal is to reduce

    theprimitive pain level

  • Abstract goalsare

    created if they satisfy

    the primitive goals


Goal Creation Experiment

Sensory-motor pairs and their effect on the environment


Goal Creation Experiment in ML

Pain signals in GCS simulation


Goal Creation Experiment in ML

Action scatters in 5 GCS simulations


Goal Creation Experiment in ML

The average pain signals in 100 GCS simulations


Compare rl tdf and ml gcs

  • Mean primitive pain Pp value as a function of the number of iterations:

    • - green line for TDF

    • blue line for GCS.

  • Primitive pain ratio with pain threshold 0.1

Compare RL (TDF) and ML (GCS)


Compare rl tdf and ml gcs1

Problem solved

Compare RL (TDF) and ML (GCS)

  • Comparison of execution time on log-log scale

    • TD-Falcon green

    • GCS blue

  • Combined efficiency of GCS 1000 better than TDF

Conclusion: embodied intelligence, with motivated learning based on goal creation is an effective learning and decision making system for dynamic environments.


Reinforcement learning motivated learning

Single value function

Measurable rewards

Can be optimized

Predictable

Objectives set by designer

Maximizes the reward

Potentially unstable

Learning effort increases with complexity

Always active

Multiple value functions

One for each goal

Internal rewards

Cannot be optimized

Unpredictable

Sets its own objectives

Solves minimax problem

Always stable

Learns better in complex environment than RL

Acts when needed

Reinforcement Learning Motivated Learning


Sounds like science fiction
Sounds like science fiction

  • If you’re trying to look far ahead, and what you see seems like science fiction, it might be wrong.

  • But if it doesn’t seem like science fiction, it’s definitely wrong.

    From presentation by Feresight Institute



Resources evolution of electronics
Resources – Evolution of Electronics

From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006



Clock Speed (doubles every 2.7 years)

From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006


Doubling (or Halving) times

  • Dynamic RAM Memory “Half Pitch” Feature Size 5.4 years

  • Dynamic RAM Memory (bits per dollar) 1.5 years

  • Average Transistor Price 1.6 years

  • Microprocessor Cost per Transistor Cycle 1.1 years

  • Total Bits Shipped 1.1 years

  • Processor Performance in MIPS 1.8 years

  • Transistors in Intel Microprocessors 2.0 years

  • Microprocessor Clock Speed 2.7 years

From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006




Software or hardware

Sequential 13, 2006

Error prone

Require programming

Low cost

Well developed programming methods

Concurrent

Robust

Require design

Significant cost

Hardware prototypes hard to build

Software or hardware?

Software Hardware


Future software hardware capabilities
Future software/hardware capabilities 13, 2006

Human

brain

complexity


Why should we care
Why should we care? 13, 2006

Source: SEMATECH


Design Productivity Gap 13, 2006 Low-Value Designs?

Percent of die area that must be occupied by memory to

maintain SOC design productivity

Source = Japanese system-LSI industry


Self-Organizing Learning Arrays SOLAR 13, 2006

  • * Self-organization

  • * Sparse and local interconnections

  • * Dynamically reconfigurable

  • * Online data-driven learning

  • Integrated circuits connect transistors into a system

    • millions of transistors easily assembled

    • first 50 years of microelectronic revolution

  • Self-organizing arrays connect processors into a system

    • millions of processors easily assembled

    • next 50 years of microelectronic revolution


ad