CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN - PowerPoint PPT Presentation

Conceptual knowledge evidence from corpora the mind and the brain l.jpg
Download
1 / 77

  • 285 Views
  • Updated On :
  • Presentation posted in: Pets / Animals

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN. Massimo Poesio Uni Trento, Center for Mind / Brain Sciences Uni Essex, Language & Computation (joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy). MOTIVATIONS.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Conceptual knowledge evidence from corpora the mind and the brain l.jpg

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN

Massimo PoesioUni Trento, Center for Mind / Brain SciencesUni Essex, Language & Computation(joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy)


Motivations l.jpg

MOTIVATIONS

  • Research on conceptual knowledge is carried out in Computational Linguistics, Neural Science, and Psychology

  • But there is limited interchange between CL and the other disciplines studying concepts

    • Except indirectly through the use of WordNet

  • This work: use data from Psychology and Neural Science to evaluate (vector-space) models produced in CL


Outline l.jpg

OUTLINE

  • Vector space representations

  • A `semantic’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


Conceptual semantics in vector space l.jpg

CONCEPTUAL SEMANTICS IN VECTOR SPACE


Lexical acquisition in corpus comp ling l.jpg

LEXICAL ACQUISITION IN CORPUS / COMP LING

  • Vectorial representations of lexical meaning derived from IR

  • WORD-BASED vector models:

    • vector dimensions are words

    • Schuetze 91, 98; HAL, LSA, Turney, Rapp

  • GRAMMATICAL RELATION models:

    • vector dimensions are pairs <Rel,W>

    • Grefenstette 93, Lin 98, Curran&Moens, Pantel, Widdows, Pado & Lapata, …..


Features in vector space models l.jpg

FEATURES IN VECTOR SPACE MODELS

GRAMMATICAL RELATIONS

WORDS


Strenghts of this approach categorization l.jpg

STRENGHTS OF THIS APPROACH: CATEGORIZATION


Limitations of this work l.jpg

LIMITATIONS OF THIS WORK

  • Very simplistic view of concepts

    • In fact, typically extract lexical representations for WORDS (non-disambiguated)

  • Limited evaluation

    • Typical evaluation: judges’ opinions about correctness of distances / comparing with WordNet

  • Most work not connected with work on concepts in Psychology / Neural Science


Our work l.jpg

OUR WORK

  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques

  • Develop task-based methods for evaluating the results

  • Integrate results from corpora with results from psychology & neural science


This talk l.jpg

THIS TALK

  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques

  • Develop task-based methods for evaluating the results

  • Integrate results from corpora with results from psychology & neural science


Outline11 l.jpg

OUTLINE

  • Vector space representations

  • A `semantic’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


More advanced theories of concepts l.jpg

MORE ADVANCED THEORIES OF CONCEPTS

  • In Linguistics:

    • Pustejovsky

  • In AI:

    • Description Logics

    • Formal ontologies

  • In Psychology:

    • Theory Theory (Murphy, 2002)

    • FUSS (Vigliocco Vinson et al)


Semantic concept descriptions pustejovsky 1991 1995 l.jpg

SEMANTIC CONCEPT DESCRIPTIONSPUSTEJOVSKY (1991, 1995)

  • Lexical entries have a QUALIA STRUCTURE consisting of four ‘roles’

    • FORMAL role: what type of object it is (shape, color, ….)

    • CONSTITUTIVE role: what it consists of (parts, stuff, etc.)

      • E.g., for books, chapters, index, paper ….

    • TELIC role: what is the purpose of the object (e.g., for books, READING)

    • AGENTIVE role: how the object was created (e.g., for books, WRITING)


Beyond bundles of attributes description logics theory theory l.jpg

BEYOND BUNDLES OF ATTRIBUTES: DESCRIPTION LOGICS, THEORY THEORY

  • We know much more about concepts than the fact that they have certain attributes:

    • We know that cars have 4 wheels whereas bicycles have 2

    • We don’t just know that people have heads, bodies and legs, but that heads are attached in certain positions whereas legs are attached in other ones

    • Facts of this type can be expressed even in the simplest concept description languages, those of description logics


Beyond simple relations description logics l.jpg

BEYOND SIMPLE RELATIONS: DESCRIPTION LOGICS

Bear  (and Animal ( 4 Paw) …)

Strawberry  (and Fruit (fills Color red) … )

Female  (and Human ( Male))


Word sense discrimination l.jpg

WORD SENSE DISCRIMINATION

  • The senses of palm in WordNet

    • the inner surface of the hand from the wrist to the base of the fingers

    • a linear unit based on the length or width of the human hand

    • any plant of the family Palmae having an unbranched trunk crowned by large pinnate or palmate leaves

    • an award for winning a championship or commemorating some other event


Concept acquisition meets relation extraction l.jpg

CONCEPT ACQUISITION MEETS RELATION EXTRACTION

  • We developed methods to identify SEMANTIC properties of concepts (`Deep lexical relations’)

    • ATTRIBUTES and their VALUES

      • Almuhareb & Poesio 2004, 2005

    • Extracting QUALIA

      • Poesio & Almuhareb 2005

    • Letting relations emerge from the data: STRUDEL

      • Baroni et al, Cognitive Science to appear

    • Extracting Wu & Barsalou –style relations

      • Poesio Barbu Giuliano & Romano, 2008

We showed that for a variety of tasks such conceptual descriptions are ‘better’ than word-based or grammatical function-based descriptions


Almuhareb poesio 2005 using a parser l.jpg

ALMUHAREB & POESIO 2005: USING A PARSER

LOOKING ONLY FOR (POTENTIAL) ATTRIBUTES AND THEIR VALUES BETTER THAN USING ALL GRS

EVEN IF ATTRIBUTES OBTAINED USING TEXT PATTERNS (“THE X OF THE Y” )


Attributes and values vs all corpus features l.jpg

ATTRIBUTES AND VALUES VS. ALL CORPUS FEATURES


Supervised extraction of concept descriptions l.jpg

SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS

  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)

  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)


Supervised extraction of concept descriptions21 l.jpg

SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS

  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)

  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)


The classification scheme for attributes of poesio almuhareb 2005 l.jpg

THE CLASSIFICATION SCHEME FOR ATTRIBUTES OF POESIO & ALMUHAREB 2005

  • PART

    • (cfr. Guarino’s non-relational attributes, Pustejovsky’s constitutive roles)

  • RELATED OBJECT

    • Non-relational attributes other than parts, relational roles

  • QUALITY

    • Guarino’s qualities, Pustejovsky’s formal roles

  • ACTIVITY

    • Pustejosvky’s telic and agentive roles

  • RELATED AGENT

  • NOT AN ATTRIBUTE (= everything else)


A supervised feature classifier l.jpg

A SUPERVISED FEATURE CLASSIFIER

  • We developed a supervised feature classifier that relies on 4 types of information

    • Morphological info (Dixon, 1991)

    • Question patterns

    • Features of features

    • Feature use

      • Some nouns used more commonly as features than as concepts: i.e., “the F of the C is” more frequent than “the * of the F is”

  • (These last four methods all rely on info extracted from the Web)


The experiment l.jpg

THE EXPERIMENT

  • We created a BALANCED DATASET

    • ~ 400 concepts

    • representing all 21 WordNet classes, including both ABSTRACT and CONCRETE concepts

    • balanced as to ambiguity and frequency

  • We collected from the Web 20,000 candidate features of these concepts using patterns

  • We hand-classified 1,155 candidate features

  • We used these data to train

    • A binary classifier (feature / non feature)

    • A 5-way classifier


Outline25 l.jpg

OUTLINE

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • Evaluating such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


Evaluation l.jpg

EVALUATION

  • Qualitative:

    • Visual inspection

    • Ask subjects to assess correctness of the classification of the attributes

  • Quantitative:

    • Use conceptual descriptions for CLUSTERING (CATEGORIZATION)


Visual evaluation top 400 features of deer according to our classifier l.jpg

VISUAL EVALUATION: TOP 400 FEATURES OF DEER ACCORDING TO OUR CLASSIFIER


Visual evaluation qualities l.jpg

VISUAL EVALUATION: QUALITIES


Quantitative evaluation l.jpg

QUANTITATIVE EVALUATION

  • ATTRIBUTES

    • PROBLEM: can’t compare against WordNet

    • Precision / recall against hand-annotated datasets

    • Human judges (ourselves):

      • We used the classifiers to classify the top 20 features of 21 randomly chosen concepts

      • We separately evaluated the results

  • CATEGORIES:

    • Clustering of the balanced dataset

    • PROBLEM: The WordNet category structure is highly subjective


Attribute classification l.jpg

ATTRIBUTE CLASSIFICATION


Clustering with 2 way classifier l.jpg

CLUSTERING WITH 2-WAY CLASSIFIER


Clustering error analysis l.jpg

CLUSTERING: ERROR ANALYSIS


Clustering error analysis33 l.jpg

CLUSTERING: ERROR ANALYSIS


Clustering error analysis34 l.jpg

CLUSTERING: ERROR ANALYSIS


Clustering error analysis35 l.jpg

CLUSTERING: ERROR ANALYSIS

IN WORDNET: PAIN


Limits of this type of evaluation l.jpg

LIMITS OF THIS TYPE OF EVALUATION

  • No way of telling how complete / accurate are our concept descriptions

    • Both in terms of relations and in terms of their relative importance

  • No way of telling whether the category distinctions we get from WordNet are empirically founded


Beyond judges evaluation against wordnet l.jpg

BEYOND JUDGES / EVALUATION AGAINST WORDNET

  • Task-based evaluation

  • Evidence from other areas of cognitive science

  • (ESSLLI 2008 Workshop - Baroni / Evert / Lenci )


Task based black box evaluation l.jpg

TASK-BASED (BLACK-BOX) EVALUATION

  • Tasks requiring lexical knowledge:

    • Lexical tests:

      • TOEFL test (Rapp 2001, Turney 2005)

    • NLP tasks:

      • Eg, anaphora resolution (Poesio et al 2004)

    • Actual applications

      • E.g., language models (Mitchell & Lapata ACL 2009, Lapata invited talk)


Evidence from other areas of cognitive science l.jpg

EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCE

  • Attributes: evidence from psychology

    • Association lists (priming)

      • E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008)

      • Comparison against feature norms: Schulte im Walde, 2008)

    • Feature norms

  • Category distinctions: evidence from neural science


Outline40 l.jpg

OUTLINE

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


Feature based representations in psychology l.jpg

FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGY

  • Feature-based concept representations assumed by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)

  • Underpin development of prototype theory (Rosch et al)

  • Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995)

  • Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)


Feature norms l.jpg

FEATURE NORMS

  • Subjects produce lists of features for a concept

  • Weighed by number of subjects that produce them

  • Several existing (Rosch and Mervis, Garrard et al, McRae et al, Vinson and Vigliocco)

  • Substantial differences in collection methodology and results


Speaker generated features vinson and vigliocco l.jpg

SPEAKER-GENERATED FEATURES (VINSON AND VIGLIOCCO)


Comparing corpus features with feature norms almuhareb et al 2005 poesio et al 2007 l.jpg

COMPARING CORPUS FEATURES WITH FEATURE NORMS (Almuhareb et al 2005, Poesio et al 2007)

  • 35 concepts in common between the Almuhareb & Poesio dataset and the dataset produced by Vinson and Vigliocco (2002, 2003)

    • ANIMALS: bear, camel, cat, cow, dog, elephant, horse, lion, mouse, sheep, tiger, zebra

    • FRUIT: apple, banana, cherry, grape, lemon, orange, peach, pear, pineapple, strawberry, watermelon

    • VEHICLE: airplane, bicycle, boat, car, helicopter, motorcycle, ship, truck, van

  • We compared the features we obtained for these concepts with the speaker-generated features collected by Vinson and Vigliocco


Results l.jpg

RESULTS

  • Best recall: ~ 52% (using all attributes and values)

  • Best precision: ~ 19%

  • But: high correlation (ro=.777) between the distances between concept representations obtained from corpora, and the distances between the representations for the same concepts obtained from subjects (using the cosine as a measure of similarity)


Discussion l.jpg

DISCUSSION

  • Substantial differences in features and overlap, but correlation similar

  • Problems:

    • Each feature norm slightly different

    • They have been normalized by hand: LOUD, NOISY, NOISE all mapped to LOUD


An example strawberry l.jpg

AN EXAMPLE: STRAWBERRY


Problem differences between feature norms l.jpg

Problem: differences between feature norms

  • motorcycle

    • Vinson & Vigliocco:

      • wheel, motor, loud, vehicle, wheel, fast, handle, ride, transport, bike, human, danger, noise, seat, brake, drive, fun, gas, machine, object, open, small, travel, wind

    • Garrard et al:

      • vehicle, wheel, fast, handlebar, light, seat, make a noise, tank, metal, unstable, tyre, coloured, sidecar, indicator, pannier, pedal, speedometer, manoeuvrable, race, brakes, stop, move, engine, petrol, economical, gears

    • McRae et al:

      • wheels, 2_wheels, dangerous, engine, fast, helmets, Harley_Davidson, loud, 1_or_2_people, vehicle, leather, transportation, 2_people, fun, Hell's_Angels, gasoline

    • Mutual correlation of ranks ranges from 0.4 to 0.7


Discussion49 l.jpg

DISCUSSION

  • Preliminary conclusions: need to collect new feature norms for CL

    • E.g., use similar techniques to collect attributes for WordNet

    • See Kremer & Baroni 2008

  • For more work on using feature norms for conceptual acquisition, see

    • Schulte im Walde 2008

    • Baroni et al to appear

  • For the correlation between feature norms and information in WordNet (meronymy, isa, plus info from glosses): Barbu & Poesio GWC 2008


Outline50 l.jpg

OUTLINE

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and brain data


Using brain data to identify category distinctions l.jpg

USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONS

  • Studies of brain-damaged patients have been shown to provide useful insights in the organization of conceptual knowledge in the brain

    • Warrington and Shallice 1984, Caramazza & Shilton 1998

  • fMRI has been used to identify these distinctions in healthy patients as well

    • E.g., Martin & Chao

  • See, e.g., Capitani et al 2003 for a survey


Category distinctions in the brain l.jpg

ANIMALS

TOOLS

CATEGORY DISTINCTIONS IN THE BRAIN


Corpus data and brain data l.jpg

CORPUS DATA AND BRAIN DATA

  • Can brain data (from healthy patients) be used to get an ‘objective’ picture of categorical distinctions in the brain?

  • Can our findings be useful to understand better the neurological results?

  • Ongoing project: using EEG and fMRI to identify such distinctions


Eeg spectral analysis of concepts l.jpg

EEG Spectral Analysis of Concepts

  • Participants presented with aural or visual concept stimuli

  • EEG apparatus records electrical activity on the scalp

  • Waveforms can be reduced to frequency components


Eeg vs fmri l.jpg

EEG vs. fMRI


Eeg pros and cons l.jpg

EEG pros and cons

  • Pros:

    • Lighter

    • Cheaper

    • Better temporal resolution (ms)

  • Cons:

    • Coarser spatial resolution (cm)

    • Noisy (e.g., very sensitive to skull depth)


A category distinction experiment with eeg l.jpg

A CATEGORY DISTINCTION EXPERIMENT WITH EEG

  • Murphy Poesio Bovolo DalPonte & Bruzzone, Cogsci 2008

  • Seven Italian native speakers

  • Image stimuli only:

    • 30 tools

    • 30 animals

  • Each stimulus presented six times

  • Optimal time / frequency window identified automatically

    • 100-370ms

    • 3-17 Hz


Stimuli images from web l.jpg

Stimuli: Images from Web


Data analysis l.jpg

Data analysis

  • Preprocessing

    • Artefact removal

  • Feature extraction

    • CSSD: a form of supervised component analysis

  • Classification

    • Using SVMs


Eeg signals time frequency per channel l.jpg

EEG SIGNALS: TIME-FREQUENCY (PER CHANNEL)


Extraction of features eegoxels from eeg data l.jpg

Extraction of features (EEGOXELS) from EEG data


Data analysis62 l.jpg

Classification System Schematic

64 channels

preprocessed data

X channels

filtered data*

“Tool” component

Feature vector

Filter by Time, Freq and Eelectr.

CSSD Decomposition

Vector Transform

var(“tool”), var(“animal”)

SupVec Machine

Answer

?

“Animal” component

Data analysis


Results63 l.jpg

RESULTS


Representation of categories in cssd spaces l.jpg

Representation of categories in CSSD spaces

  • component analysis identifies 2-dimensional spaces

  • Analysis of these spaces may provide useful data against which to compare our corpus models


Cssd derived conceptual spaces l.jpg

CSSD-derived conceptual spaces


Brain data and corpus data l.jpg

BRAIN DATA AND CORPUS DATA

  • What is the relation between the conceptual spaces induced from corpora with the conceptual spaces elicited using EEG?


Predicting brain fmri activation using concept descriptions l.jpg

PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONS

  • T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science320, 1191–1195


Mitchell et al 2008 methods l.jpg

MITCHELL ET AL 2008: METHODS

  • Record fMRI activation for 60 nominal concepts

    • And extract 200 ‘best’ features, or VOXELs

  • Build conceptual descriptions for these concepts from corpora (the Web)

    • 25 features for each concept

    • 25 verbs expressing typical properties of living things / tools

    • Collect strength of association between these features and each concept

  • Learn association between each voxel and the 25 verbal features using 58 concepts

  • Use learned model to predict activation of 2 held-out data (compare using Euclidean distance)

    • Accuracy: 77%


Mitchell et al 2008 l.jpg

MITCHELL ET AL 2008


Mitchell et al 2008 verb features l.jpg

MITCHELL ET AL 2008: VERB FEATURES


Mitchell et al learning associations l.jpg

MITCHELL ET AL: LEARNING ASSOCIATIONS


Our experiments l.jpg

OUR EXPERIMENTS

  • Replicate the Mitchell et al study using EEG data instead of fMRI

    • Different feature selection mechanisms

  • Compare different methods for building concept descriptions

    • In addition to hand-picked, also a variety of standard corpus models

  • For Italian

  • B. Murphy, M. Baroni, M. Poesio, EEG responds to conceptual stimuli and corpus semantics, EMNLP 2009


Results using the hand picked features l.jpg

RESULTS USING THE HAND-PICKED FEATURES


Results using automatically selected features l.jpg

AA-MP

MITCHELL ET AL

RESULTS USING AUTOMATICALLY SELECTED FEATURES


Recap l.jpg

RECAP

  • We need to relate the evidence from corpora with evidence about concepts coming from empirical work in the neuroscience and psychology

  • Feature norms databases could be used to evaluate attribute extraction

    • But: need to find better ways of collecting them

  • Brain data may give us information about the ‘real’ conceptual categories

    • Results still preliminary


Collaborators l.jpg

COLLABORATORS

MARCO BARONI(Trento)

BRIAN MURPHY(Trento)

ABDULRAHMAN ALMUHAREB(Essex PhD 2006 Now at KACST, Saudi Arabia)

EDUARD BARBU(Trento PhD forthc)

HEBA LAKANY(formerly Essex, now Strathclyde)


Thanks l.jpg

THANKS

To the audience …..

And to Galja, Ruslan & the other organizers for yet another splendid RANLP!


  • Login