Conceptual knowledge evidence from corpora the mind and the brain
Download
1 / 77

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA - PowerPoint PPT Presentation


  • 303 Views
  • Updated On :

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN. Massimo Poesio Uni Trento, Center for Mind / Brain Sciences Uni Essex, Language & Computation (joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy). MOTIVATIONS.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA' - arleen


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Conceptual knowledge evidence from corpora the mind and the brain l.jpg

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN

Massimo PoesioUni Trento, Center for Mind / Brain SciencesUni Essex, Language & Computation(joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy)


Motivations l.jpg
MOTIVATIONS THE BRAIN

  • Research on conceptual knowledge is carried out in Computational Linguistics, Neural Science, and Psychology

  • But there is limited interchange between CL and the other disciplines studying concepts

    • Except indirectly through the use of WordNet

  • This work: use data from Psychology and Neural Science to evaluate (vector-space) models produced in CL


Outline l.jpg
OUTLINE THE BRAIN

  • Vector space representations

  • A `semantic’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data



Lexical acquisition in corpus comp ling l.jpg
LEXICAL ACQUISITION IN CORPUS / COMP LING THE BRAIN

  • Vectorial representations of lexical meaning derived from IR

  • WORD-BASED vector models:

    • vector dimensions are words

    • Schuetze 91, 98; HAL, LSA, Turney, Rapp

  • GRAMMATICAL RELATION models:

    • vector dimensions are pairs <Rel,W>

    • Grefenstette 93, Lin 98, Curran&Moens, Pantel, Widdows, Pado & Lapata, …..


Features in vector space models l.jpg
FEATURES IN VECTOR SPACE MODELS THE BRAIN

GRAMMATICAL RELATIONS

WORDS



Limitations of this work l.jpg
LIMITATIONS OF THIS WORK THE BRAIN

  • Very simplistic view of concepts

    • In fact, typically extract lexical representations for WORDS (non-disambiguated)

  • Limited evaluation

    • Typical evaluation: judges’ opinions about correctness of distances / comparing with WordNet

  • Most work not connected with work on concepts in Psychology / Neural Science


Our work l.jpg
OUR WORK THE BRAIN

  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques

  • Develop task-based methods for evaluating the results

  • Integrate results from corpora with results from psychology & neural science


This talk l.jpg
THIS TALK THE BRAIN

  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques

  • Develop task-based methods for evaluating the results

  • Integrate results from corpora with results from psychology & neural science


Outline11 l.jpg
OUTLINE THE BRAIN

  • Vector space representations

  • A `semantic’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


More advanced theories of concepts l.jpg
MORE ADVANCED THEORIES OF CONCEPTS THE BRAIN

  • In Linguistics:

    • Pustejovsky

  • In AI:

    • Description Logics

    • Formal ontologies

  • In Psychology:

    • Theory Theory (Murphy, 2002)

    • FUSS (Vigliocco Vinson et al)


Semantic concept descriptions pustejovsky 1991 1995 l.jpg
SEMANTIC CONCEPT DESCRIPTIONS THE BRAINPUSTEJOVSKY (1991, 1995)

  • Lexical entries have a QUALIA STRUCTURE consisting of four ‘roles’

    • FORMAL role: what type of object it is (shape, color, ….)

    • CONSTITUTIVE role: what it consists of (parts, stuff, etc.)

      • E.g., for books, chapters, index, paper ….

    • TELIC role: what is the purpose of the object (e.g., for books, READING)

    • AGENTIVE role: how the object was created (e.g., for books, WRITING)


Beyond bundles of attributes description logics theory theory l.jpg
BEYOND BUNDLES OF ATTRIBUTES: THE BRAINDESCRIPTION LOGICS, THEORY THEORY

  • We know much more about concepts than the fact that they have certain attributes:

    • We know that cars have 4 wheels whereas bicycles have 2

    • We don’t just know that people have heads, bodies and legs, but that heads are attached in certain positions whereas legs are attached in other ones

    • Facts of this type can be expressed even in the simplest concept description languages, those of description logics


Beyond simple relations description logics l.jpg
BEYOND SIMPLE RELATIONS: DESCRIPTION LOGICS THE BRAIN

Bear  (and Animal ( 4 Paw) …)

Strawberry  (and Fruit (fills Color red) … )

Female  (and Human ( Male))


Word sense discrimination l.jpg
WORD SENSE DISCRIMINATION THE BRAIN

  • The senses of palm in WordNet

    • the inner surface of the hand from the wrist to the base of the fingers

    • a linear unit based on the length or width of the human hand

    • any plant of the family Palmae having an unbranched trunk crowned by large pinnate or palmate leaves

    • an award for winning a championship or commemorating some other event


Concept acquisition meets relation extraction l.jpg
CONCEPT ACQUISITION MEETS RELATION EXTRACTION THE BRAIN

  • We developed methods to identify SEMANTIC properties of concepts (`Deep lexical relations’)

    • ATTRIBUTES and their VALUES

      • Almuhareb & Poesio 2004, 2005

    • Extracting QUALIA

      • Poesio & Almuhareb 2005

    • Letting relations emerge from the data: STRUDEL

      • Baroni et al, Cognitive Science to appear

    • Extracting Wu & Barsalou –style relations

      • Poesio Barbu Giuliano & Romano, 2008

We showed that for a variety of tasks such conceptual descriptions are ‘better’ than word-based or grammatical function-based descriptions


Almuhareb poesio 2005 using a parser l.jpg
ALMUHAREB & POESIO 2005: THE BRAINUSING A PARSER

LOOKING ONLY FOR (POTENTIAL) ATTRIBUTES AND THEIR VALUES BETTER THAN USING ALL GRS

EVEN IF ATTRIBUTES OBTAINED USING TEXT PATTERNS (“THE X OF THE Y” )



Supervised extraction of concept descriptions l.jpg
SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS THE BRAIN

  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)

  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)


Supervised extraction of concept descriptions21 l.jpg
SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS THE BRAIN

  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)

  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)


The classification scheme for attributes of poesio almuhareb 2005 l.jpg
THE CLASSIFICATION SCHEME FOR ATTRIBUTES OF POESIO & ALMUHAREB 2005

  • PART

    • (cfr. Guarino’s non-relational attributes, Pustejovsky’s constitutive roles)

  • RELATED OBJECT

    • Non-relational attributes other than parts, relational roles

  • QUALITY

    • Guarino’s qualities, Pustejovsky’s formal roles

  • ACTIVITY

    • Pustejosvky’s telic and agentive roles

  • RELATED AGENT

  • NOT AN ATTRIBUTE (= everything else)


A supervised feature classifier l.jpg
A SUPERVISED FEATURE CLASSIFIER ALMUHAREB 2005

  • We developed a supervised feature classifier that relies on 4 types of information

    • Morphological info (Dixon, 1991)

    • Question patterns

    • Features of features

    • Feature use

      • Some nouns used more commonly as features than as concepts: i.e., “the F of the C is” more frequent than “the * of the F is”

  • (These last four methods all rely on info extracted from the Web)


The experiment l.jpg
THE EXPERIMENT ALMUHAREB 2005

  • We created a BALANCED DATASET

    • ~ 400 concepts

    • representing all 21 WordNet classes, including both ABSTRACT and CONCRETE concepts

    • balanced as to ambiguity and frequency

  • We collected from the Web 20,000 candidate features of these concepts using patterns

  • We hand-classified 1,155 candidate features

  • We used these data to train

    • A binary classifier (feature / non feature)

    • A 5-way classifier


Outline25 l.jpg
OUTLINE ALMUHAREB 2005

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • Evaluating such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


Evaluation l.jpg
EVALUATION ALMUHAREB 2005

  • Qualitative:

    • Visual inspection

    • Ask subjects to assess correctness of the classification of the attributes

  • Quantitative:

    • Use conceptual descriptions for CLUSTERING (CATEGORIZATION)


Visual evaluation top 400 features of deer according to our classifier l.jpg
VISUAL EVALUATION: ALMUHAREB 2005TOP 400 FEATURES OF DEER ACCORDING TO OUR CLASSIFIER



Quantitative evaluation l.jpg
QUANTITATIVE EVALUATION ALMUHAREB 2005

  • ATTRIBUTES

    • PROBLEM: can’t compare against WordNet

    • Precision / recall against hand-annotated datasets

    • Human judges (ourselves):

      • We used the classifiers to classify the top 20 features of 21 randomly chosen concepts

      • We separately evaluated the results

  • CATEGORIES:

    • Clustering of the balanced dataset

    • PROBLEM: The WordNet category structure is highly subjective


Attribute classification l.jpg
ATTRIBUTE CLASSIFICATION ALMUHAREB 2005






Clustering error analysis35 l.jpg
CLUSTERING: ERROR ANALYSIS ALMUHAREB 2005

IN WORDNET: PAIN


Limits of this type of evaluation l.jpg
LIMITS OF THIS TYPE OF EVALUATION ALMUHAREB 2005

  • No way of telling how complete / accurate are our concept descriptions

    • Both in terms of relations and in terms of their relative importance

  • No way of telling whether the category distinctions we get from WordNet are empirically founded


Beyond judges evaluation against wordnet l.jpg
BEYOND JUDGES / EVALUATION AGAINST WORDNET ALMUHAREB 2005

  • Task-based evaluation

  • Evidence from other areas of cognitive science

  • (ESSLLI 2008 Workshop - Baroni / Evert / Lenci )


Task based black box evaluation l.jpg
TASK-BASED (BLACK-BOX) EVALUATION ALMUHAREB 2005

  • Tasks requiring lexical knowledge:

    • Lexical tests:

      • TOEFL test (Rapp 2001, Turney 2005)

    • NLP tasks:

      • Eg, anaphora resolution (Poesio et al 2004)

    • Actual applications

      • E.g., language models (Mitchell & Lapata ACL 2009, Lapata invited talk)


Evidence from other areas of cognitive science l.jpg
EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCE ALMUHAREB 2005

  • Attributes: evidence from psychology

    • Association lists (priming)

      • E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008)

      • Comparison against feature norms: Schulte im Walde, 2008)

    • Feature norms

  • Category distinctions: evidence from neural science


Outline40 l.jpg
OUTLINE ALMUHAREB 2005

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and Brain data


Feature based representations in psychology l.jpg
FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGY ALMUHAREB 2005

  • Feature-based concept representations assumed by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)

  • Underpin development of prototype theory (Rosch et al)

  • Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995)

  • Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)


Feature norms l.jpg
FEATURE NORMS ALMUHAREB 2005

  • Subjects produce lists of features for a concept

  • Weighed by number of subjects that produce them

  • Several existing (Rosch and Mervis, Garrard et al, McRae et al, Vinson and Vigliocco)

  • Substantial differences in collection methodology and results



Comparing corpus features with feature norms almuhareb et al 2005 poesio et al 2007 l.jpg
COMPARING CORPUS FEATURES WITH FEATURE NORMS ALMUHAREB 2005(Almuhareb et al 2005, Poesio et al 2007)

  • 35 concepts in common between the Almuhareb & Poesio dataset and the dataset produced by Vinson and Vigliocco (2002, 2003)

    • ANIMALS: bear, camel, cat, cow, dog, elephant, horse, lion, mouse, sheep, tiger, zebra

    • FRUIT: apple, banana, cherry, grape, lemon, orange, peach, pear, pineapple, strawberry, watermelon

    • VEHICLE: airplane, bicycle, boat, car, helicopter, motorcycle, ship, truck, van

  • We compared the features we obtained for these concepts with the speaker-generated features collected by Vinson and Vigliocco


Results l.jpg
RESULTS ALMUHAREB 2005

  • Best recall: ~ 52% (using all attributes and values)

  • Best precision: ~ 19%

  • But: high correlation (ro=.777) between the distances between concept representations obtained from corpora, and the distances between the representations for the same concepts obtained from subjects (using the cosine as a measure of similarity)


Discussion l.jpg
DISCUSSION ALMUHAREB 2005

  • Substantial differences in features and overlap, but correlation similar

  • Problems:

    • Each feature norm slightly different

    • They have been normalized by hand: LOUD, NOISY, NOISE all mapped to LOUD


An example strawberry l.jpg
AN EXAMPLE: STRAWBERRY ALMUHAREB 2005


Problem differences between feature norms l.jpg
Problem: differences between feature norms ALMUHAREB 2005

  • motorcycle

    • Vinson & Vigliocco:

      • wheel, motor, loud, vehicle, wheel, fast, handle, ride, transport, bike, human, danger, noise, seat, brake, drive, fun, gas, machine, object, open, small, travel, wind

    • Garrard et al:

      • vehicle, wheel, fast, handlebar, light, seat, make a noise, tank, metal, unstable, tyre, coloured, sidecar, indicator, pannier, pedal, speedometer, manoeuvrable, race, brakes, stop, move, engine, petrol, economical, gears

    • McRae et al:

      • wheels, 2_wheels, dangerous, engine, fast, helmets, Harley_Davidson, loud, 1_or_2_people, vehicle, leather, transportation, 2_people, fun, Hell's_Angels, gasoline

    • Mutual correlation of ranks ranges from 0.4 to 0.7


Discussion49 l.jpg
DISCUSSION ALMUHAREB 2005

  • Preliminary conclusions: need to collect new feature norms for CL

    • E.g., use similar techniques to collect attributes for WordNet

    • See Kremer & Baroni 2008

  • For more work on using feature norms for conceptual acquisition, see

    • Schulte im Walde 2008

    • Baroni et al to appear

  • For the correlation between feature norms and information in WordNet (meronymy, isa, plus info from glosses): Barbu & Poesio GWC 2008


Outline50 l.jpg
OUTLINE ALMUHAREB 2005

  • Vector space representations

  • An example of `Semantic-based’ vector space model

  • How to evaluate such models

  • Attribute extraction and Feature norms

  • Category distinctions and brain data


Using brain data to identify category distinctions l.jpg
USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONS ALMUHAREB 2005

  • Studies of brain-damaged patients have been shown to provide useful insights in the organization of conceptual knowledge in the brain

    • Warrington and Shallice 1984, Caramazza & Shilton 1998

  • fMRI has been used to identify these distinctions in healthy patients as well

    • E.g., Martin & Chao

  • See, e.g., Capitani et al 2003 for a survey


Category distinctions in the brain l.jpg

ANIMALS ALMUHAREB 2005

TOOLS

CATEGORY DISTINCTIONS IN THE BRAIN


Corpus data and brain data l.jpg
CORPUS DATA AND BRAIN DATA ALMUHAREB 2005

  • Can brain data (from healthy patients) be used to get an ‘objective’ picture of categorical distinctions in the brain?

  • Can our findings be useful to understand better the neurological results?

  • Ongoing project: using EEG and fMRI to identify such distinctions


Eeg spectral analysis of concepts l.jpg
EEG Spectral Analysis of Concepts ALMUHAREB 2005

  • Participants presented with aural or visual concept stimuli

  • EEG apparatus records electrical activity on the scalp

  • Waveforms can be reduced to frequency components


Eeg vs fmri l.jpg
EEG vs. fMRI ALMUHAREB 2005


Eeg pros and cons l.jpg
EEG pros and cons ALMUHAREB 2005

  • Pros:

    • Lighter

    • Cheaper

    • Better temporal resolution (ms)

  • Cons:

    • Coarser spatial resolution (cm)

    • Noisy (e.g., very sensitive to skull depth)


A category distinction experiment with eeg l.jpg
A CATEGORY DISTINCTION EXPERIMENT WITH EEG ALMUHAREB 2005

  • Murphy Poesio Bovolo DalPonte & Bruzzone, Cogsci 2008

  • Seven Italian native speakers

  • Image stimuli only:

    • 30 tools

    • 30 animals

  • Each stimulus presented six times

  • Optimal time / frequency window identified automatically

    • 100-370ms

    • 3-17 Hz


Stimuli images from web l.jpg
Stimuli: Images from Web ALMUHAREB 2005


Data analysis l.jpg
Data analysis ALMUHAREB 2005

  • Preprocessing

    • Artefact removal

  • Feature extraction

    • CSSD: a form of supervised component analysis

  • Classification

    • Using SVMs


Eeg signals time frequency per channel l.jpg
EEG SIGNALS: ALMUHAREB 2005TIME-FREQUENCY (PER CHANNEL)



Data analysis62 l.jpg

Classification System Schematic ALMUHAREB 2005

64 channels

preprocessed data

X channels

filtered data*

“Tool” component

Feature vector

Filter by Time, Freq and Eelectr.

CSSD Decomposition

Vector Transform

var(“tool”), var(“animal”)

SupVec Machine

Answer

?

“Animal” component

Data analysis


Results63 l.jpg
RESULTS ALMUHAREB 2005


Representation of categories in cssd spaces l.jpg
Representation of categories in CSSD spaces ALMUHAREB 2005

  • component analysis identifies 2-dimensional spaces

  • Analysis of these spaces may provide useful data against which to compare our corpus models



Brain data and corpus data l.jpg
BRAIN DATA AND CORPUS DATA ALMUHAREB 2005

  • What is the relation between the conceptual spaces induced from corpora with the conceptual spaces elicited using EEG?


Predicting brain fmri activation using concept descriptions l.jpg
PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONS

  • T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science320, 1191–1195


Mitchell et al 2008 methods l.jpg
MITCHELL ET AL 2008: METHODS DESCRIPTIONS

  • Record fMRI activation for 60 nominal concepts

    • And extract 200 ‘best’ features, or VOXELs

  • Build conceptual descriptions for these concepts from corpora (the Web)

    • 25 features for each concept

    • 25 verbs expressing typical properties of living things / tools

    • Collect strength of association between these features and each concept

  • Learn association between each voxel and the 25 verbal features using 58 concepts

  • Use learned model to predict activation of 2 held-out data (compare using Euclidean distance)

    • Accuracy: 77%


Mitchell et al 2008 l.jpg
MITCHELL ET AL 2008 DESCRIPTIONS


Mitchell et al 2008 verb features l.jpg
MITCHELL ET AL 2008: DESCRIPTIONSVERB FEATURES


Mitchell et al learning associations l.jpg
MITCHELL ET AL: DESCRIPTIONSLEARNING ASSOCIATIONS


Our experiments l.jpg
OUR EXPERIMENTS DESCRIPTIONS

  • Replicate the Mitchell et al study using EEG data instead of fMRI

    • Different feature selection mechanisms

  • Compare different methods for building concept descriptions

    • In addition to hand-picked, also a variety of standard corpus models

  • For Italian

  • B. Murphy, M. Baroni, M. Poesio, EEG responds to conceptual stimuli and corpus semantics, EMNLP 2009



Results using automatically selected features l.jpg

AA-MP DESCRIPTIONS

MITCHELL ET AL

RESULTS USING AUTOMATICALLY SELECTED FEATURES


Recap l.jpg
RECAP DESCRIPTIONS

  • We need to relate the evidence from corpora with evidence about concepts coming from empirical work in the neuroscience and psychology

  • Feature norms databases could be used to evaluate attribute extraction

    • But: need to find better ways of collecting them

  • Brain data may give us information about the ‘real’ conceptual categories

    • Results still preliminary


Collaborators l.jpg
COLLABORATORS DESCRIPTIONS

MARCO BARONI(Trento)

BRIAN MURPHY(Trento)

ABDULRAHMAN ALMUHAREB(Essex PhD 2006 Now at KACST, Saudi Arabia)

EDUARD BARBU(Trento PhD forthc)

HEBA LAKANY(formerly Essex, now Strathclyde)


Thanks l.jpg
THANKS DESCRIPTIONS

To the audience …..

And to Galja, Ruslan & the other organizers for yet another splendid RANLP!


ad