using analogy to discover the meaning of pictures l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Using Analogy to Discover the Meaning of Pictures PowerPoint Presentation
Download Presentation
Using Analogy to Discover the Meaning of Pictures

Loading in 2 Seconds...

play fullscreen
1 / 87

Using Analogy to Discover the Meaning of Pictures - PowerPoint PPT Presentation


  • 110 Views
  • Uploaded on

Using Analogy to Discover the Meaning of Pictures. Melanie Mitchell Computer Science Department Portland State University and External Professor Santa Fe Institute. An image-understanding task:. High-level perception. “Meaning”. ?. Simple Segmentation. Color, Shape, Texture.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Using Analogy to Discover the Meaning of Pictures' - brooklyn


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using analogy to discover the meaning of pictures

Using Analogy to Discover the Meaning of Pictures

Melanie Mitchell

Computer Science Department

Portland State University

and

External Professor

Santa Fe Institute

slide3

High-level perception

“Meaning”

?

Simple Segmentation

Color, Shape, Texture

Object recognition

Pattern recognition

Low-level vision

slide4

High-level perception

“Meaning”

Simple Segmentation

Color, Shape, Texture

Object recognition

Pattern recognition

Low-level vision

The “SEMANTIC GAP’

gabor filters
Gabor Filters

Gabor filter: Essentially a localized

Fourier transform in the image.

Filter has associated frequency ,

scale s, and orientation .

Response measures extent to which 

is present at orientation at scale s

centered about pixel (x,y).

s1 units gabor filters one per pixel
S1 units: Gabor filters (one per pixel)

16 scales / frequencies, 4 orientations

c1 unit maximum value of group of s1 units pooled over slightly different positions and scales
C1 unit: Maximum value of group of S1 units, pooled over slightly different positions and scales

8 scales / frequencies, 4 orientations

s2 units radial basis functions over natural image patches
S2 units: Radial Basis Functions over “Natural Image Patches”
  • Idea is that natural images contain universal, low-level features that are useful in classifying objects.
  • Randomly sample small “crops” from natural images, and feed them through S1 and C1 layers.
  • Collect a set of N patches , {Pi | i 1, ..., N}, of C1 layer from this random sample.
  • Now, with new image, a unit S2i corresponding to Pi gets input X from C1 layer, computes a radial basis function:
  • Gives “degree” to which feature Piis present in input X.
slide12

Feature vector

representing image

Support Vector Machine

classification

sample of results from poggio model
Sample of results from Poggio model

(Serre et al., 2006)

(Bileschi, 2006)

slide16

Can we use a simple ontology to answer this question?

“Dog walking”

Person

Dog

leash

holds

attached to

action

action

walking

slide18

Can we use a simple ontology to answer this question?

“Dog walking”

Person

Dog

leash

holds

attached to

Dogs

action

action

running

walking

slide20

Can we use a simple ontology to answer this question?

“Dog walking”

Person

Dog

leash

holds

attached to

Dogs

action

action

Cat

running

Iguana

walking

slide25

Can we use a simple ontology to answer this question?

“Dog walking”

Person

Dog

leash

Helicopter

Bicycle

Car

holds

attached to

Dogs

action

action

Cat

running

Iguana

walking

slide29

Dog grooming

Fanny pack

Dog walking

Gasoline

Lawn mower

Sidewalk

Beach

Stick

Inside

Runway

Sky

Helicopter

Leash

Army

Grass

Airplane

Dog

Outside

Person

Ground

Holding

Attached to

Tree

Backpack

Car

Far from

Close to

Standing

Running

Above

Left of

Walking

Track

why is image understanding hard for computers30
Why is image-understanding hard for computers?
  • It is vastly open-ended.
    • Can’t solve by feeding image’s feature vector to all known “object classifiers”; in general too many such classifiers, and they are too imperfect! (Compare with StreetScenes system.)
    • In general can’t even construct high-level“feature vector” ahead of time, since there are too many possible features and you don’t know which features are relevant.
  • Need dynamics! Need to construct “probable”, coherent, consistent, representation of picture at “recognition time”. Construction process must allow different parts of representation to influence one another dynamically.
slide31
In constructing representation, need to limit exploration of features to the most promising possibilities ― but how do you know which ones are promising without exploring them?
  • Need prior, higher-level knowledge to interact with lower-level vision in both directions (bottom-up and top-down).
  • Need to allow prior knowledge to be “fluid” – allow concepts to “slip”. Need to perceive essential similarity in the face of superficial differences (analogy-making).
  • In short, need “active symbols”: concepts with dynamic activation (relevance) that can be activated by other active symbols, spread activation to conceptual neighbors, and that can push for themselves to be instantiated in the perception of a situation.
active symbol architectures hofstadter et al

Concept network

Active Symbol Architectures(Hofstadter et al.)

“Top-down” perceptual agents (codelets)

Workspace

Temperature

“Bottom-up” perceptual agents (codelets)

slide33

Architecture of Copycat

Concept network (Slipnet)

a b c ---> a b d

i i j j k k --> ?

Perceptual and structure-building agents (codelets)

Workspace

Temperature

idealizing analogy making35
Idealizing analogy-making

abc ---> abd

ijk ---> ?

idealizing analogy making36
Idealizing analogy-making

abc ---> abd

ijk ---> ijl (replace rightmost

letter by successor)

idealizing analogy making37
Idealizing analogy-making

abc ---> abd

ijk ---> ijl (replace rightmost

letter by successor)

ijd (replace rightmost

letter by ‘d’)

idealizing analogy making38
Idealizing analogy-making

abc ---> abd

ijk ---> ijl (replace rightmost

letter by successor)

ijd (replace rightmost

letter by ‘d’)

ijk (replace all

‘c’s by ‘d’s)

idealizing analogy making39
Idealizing analogy-making

abc ---> abd

ijk ---> ijl (replace rightmost

letter by successor)

ijd (replace rightmost

letter by ‘d’)

ijk (replace all

‘c’s by ‘d’s)

abd (replace any

string by ‘abd’)

idealizing analogy making40
Idealizing analogy-making

abc ---> abd

iijjkk ---> ?

idealizing analogy making41
Idealizing analogy-making

abc ---> abd

iijjkk ---> iijjkl

Replace rightmost letter by successor

idealizing analogy making42
Idealizing analogy-making

abc ---> abd

iijjkk ---> ?

idealizing analogy making43
Idealizing analogy-making

abc ---> abd

iijjkk ---> iijjll

Replace rightmost “letter” by successor

idealizing analogy making44
Idealizing analogy-making

abc ---> abd

kji ---> ?

idealizing analogy making45
Idealizing analogy-making

abc ---> abd

kji ---> kjj

Replace rightmost letter by successor

idealizing analogy making46
Idealizing analogy-making

abc ---> abd

kji ---> ?

idealizing analogy making47
Idealizing analogy-making

abc ---> abd

kji ---> lji

Replace “rightmost” letter by successor

idealizing analogy making48
Idealizing analogy-making

abc ---> abd

kji ---> ?

idealizing analogy making49
Idealizing analogy-making

abc ---> abd

kji ---> ?

idealizing analogy making50
Idealizing analogy-making

abc ---> abd

kji ---> kjh

Replace rightmost letter by “successor”

idealizing analogy making51
Idealizing analogy-making

abc ---> abd

mrrjjj ---> ?

idealizing analogy making52
Idealizing analogy-making

abc ---> abd

mrrjjj ---> mrrjjk

Replace rightmost letter by successor

idealizing analogy making53
Idealizing analogy-making

abc ---> abd

mrrjjj ---> ?

idealizing analogy making54
Idealizing analogy-making

abc ---> abd

mrrjjj ---> ?

1 2 3

idealizing analogy making55
Idealizing analogy-making

abc ---> abd

mrrjjj ---> ?

1 2 3

1 2 4

idealizing analogy making56
Idealizing analogy-making

abc ---> abd

mrrjjj ---> mrrjjjj

1 2 3

Replace rightmost “letter” by successor

1 2 4

idealizing analogy making57
Idealizing analogy-making

abc ---> abd

xyz ---> ?

idealizing analogy making58
Idealizing analogy-making

abc ---> abd

xyz ---> xya

Replace rightmost letter by successor

idealizing analogy making59
Idealizing analogy-making

abc ---> abd

xyz ---> xya (not allowed)

Replace rightmost letter by successor

idealizing analogy making60
Idealizing analogy-making

abc ---> abd

xyz ---> ?

idealizing analogy making61
Idealizing analogy-making

abc ---> abd

xyz ---> ?

last letter in alphabet

idealizing analogy making62
Idealizing analogy-making

first letter in alphabet

abc ---> abd

xyz ---> ?

last letter in alphabet

idealizing analogy making63
Idealizing analogy-making

first letter in alphabet

abc ---> abd

xyz ---> ?

last letter in alphabet

idealizing analogy making64
Idealizing analogy-making

first letter in alphabet

abc ---> abd

xyz ---> wyz

last letter in alphabet

Replace “rightmost” letter by “successor”

idealizing analogy making65
Idealizing analogy-making

first letter in alphabet

abc ---> abd

xyz ---> wyz

last letter in alphabet

abilities needed in the letter string microworld67
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
abilities needed in the letter string microworld68
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
  • Describing objects, relations, and events at the appropriate level of abstraction
abilities needed in the letter string microworld69
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
  • Describing objects, relations, and events at the appropriate level of abstraction
  • Chunking certain elements of a situation while viewing others individually
abilities needed in the letter string microworld70
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
  • Describing objects, relations, and events at the appropriate level of abstraction
  • Chunking certain elements of a situation while viewing others individually
  • Focusing on relevant aspects and ignoring irrelevant or superficial aspects of situations
abilities needed in the letter string microworld71
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
  • Describing objects, relations, and events at the appropriate level of abstraction
  • Chunking certain elements of a situation while viewing others individually
  • Focusing on relevant aspects and ignoring irrelevant or superficial aspects of situations
  • Taking certain descriptions literally and letting others slip
abilities needed in the letter string microworld72
Abilities needed in the letter-string microworld
  • Mentally constructing a coherently structured whole out of initially unattached parts
  • Describing objects, relations, and events at the appropriate level of abstraction
  • Chunking certain elements of a situation while viewing others individually
  • Focusing on relevant aspects and ignoring irrelevant or superficial aspects of situations
  • Taking certain descriptions literally and letting others slip
  • Exploring many avenues of possible interpretations while avoiding a search through a combinatorial explosion of possibilities
the copycat program hofstadter and mitchell74
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
the copycat program hofstadter and mitchell75
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
  • Understanding and perception of similarity is built up collectively by many independent simple “agents” working in parallel
the copycat program hofstadter and mitchell76
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
  • Understanding and perception of similarity is built up collectively by many independent simple “agents” working in parallel
  • Each agent has very limited perceptual and communication abilities
the copycat program hofstadter and mitchell77
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
  • Understanding and perception of similarity is built up collectively by many independent simple “agents” working in parallel
  • Each agent has very limited perceptual and communication abilities
  • Teams of agents explore different possibilities for structures, building on what previous teams have constructed.
the copycat program hofstadter and mitchell78
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
  • Understanding and perception of similarity is built up collectively by many independent simple “agents” working in parallel
  • Each agent has very limited perceptual and communication abilities
  • Teams of agents explore different possibilities for structures, building on what previous teams have constructed.
  • The resources (agent time) allocated to a possible structure depends on its promise, as assessed dynamically as exploration proceeds.
the copycat program hofstadter and mitchell79
The Copycat program(Hofstadter and Mitchell)
  • Inspired by collective behavior in complex systems (e.g., ant colonies)
  • Understanding and perception of similarity is built up collectively by many independent simple “agents” working in parallel
  • Each agent has very limited perceptual and communication abilities
  • Teams of agents explore different possibilities for structures, building on what previous teams have constructed.
  • The resources (agent time) allocated to a possible structure depends on its promise, as assessed dynamically as exploration proceeds.
  • The agents working together produce an “emergent” understanding of the analogy.
slide80

Copycat/Metacat demo

(M. Mitchell, J. Marshall, D. Hofstadter)

acknowledgments
Acknowledgments
  • Thanks to the J. S. McDonnell Foundation, the National Science Foundation, and Portland State University for research support.
  • Thanks to all of you for listening!