conceptual knowledge evidence from corpora the mind and the brain
Download
Skip this Video
Download Presentation
CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN

Loading in 2 Seconds...

play fullscreen
1 / 77

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA - PowerPoint PPT Presentation


  • 304 Views
  • Uploaded on

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN. Massimo Poesio Uni Trento, Center for Mind / Brain Sciences Uni Essex, Language & Computation (joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy). MOTIVATIONS.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA' - arleen


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
conceptual knowledge evidence from corpora the mind and the brain

CONCEPTUAL KNOWLEDGE: EVIDENCE FROM CORPORA, THE MIND, AND THE BRAIN

Massimo PoesioUni Trento, Center for Mind / Brain SciencesUni Essex, Language & Computation(joint work with A. Almuhareb, E. Barbu, M. Baroni, B. Murphy)

motivations
MOTIVATIONS
  • Research on conceptual knowledge is carried out in Computational Linguistics, Neural Science, and Psychology
  • But there is limited interchange between CL and the other disciplines studying concepts
    • Except indirectly through the use of WordNet
  • This work: use data from Psychology and Neural Science to evaluate (vector-space) models produced in CL
outline
OUTLINE
  • Vector space representations
  • A `semantic’ vector space model
  • How to evaluate such models
  • Attribute extraction and Feature norms
  • Category distinctions and Brain data
lexical acquisition in corpus comp ling
LEXICAL ACQUISITION IN CORPUS / COMP LING
  • Vectorial representations of lexical meaning derived from IR
  • WORD-BASED vector models:
    • vector dimensions are words
    • Schuetze 91, 98; HAL, LSA, Turney, Rapp
  • GRAMMATICAL RELATION models:
    • vector dimensions are pairs <Rel,W>
    • Grefenstette 93, Lin 98, Curran&Moens, Pantel, Widdows, Pado & Lapata, …..
features in vector space models
FEATURES IN VECTOR SPACE MODELS

GRAMMATICAL RELATIONS

WORDS

limitations of this work
LIMITATIONS OF THIS WORK
  • Very simplistic view of concepts
    • In fact, typically extract lexical representations for WORDS (non-disambiguated)
  • Limited evaluation
    • Typical evaluation: judges’ opinions about correctness of distances / comparing with WordNet
  • Most work not connected with work on concepts in Psychology / Neural Science
our work
OUR WORK
  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques
  • Develop task-based methods for evaluating the results
  • Integrate results from corpora with results from psychology & neural science
this talk
THIS TALK
  • Acquire richer, more semantic-oriented concept descriptions by exploiting relation extraction techniques
  • Develop task-based methods for evaluating the results
  • Integrate results from corpora with results from psychology & neural science
outline11
OUTLINE
  • Vector space representations
  • A `semantic’ vector space model
  • How to evaluate such models
  • Attribute extraction and Feature norms
  • Category distinctions and Brain data
more advanced theories of concepts
MORE ADVANCED THEORIES OF CONCEPTS
  • In Linguistics:
    • Pustejovsky
  • In AI:
    • Description Logics
    • Formal ontologies
  • In Psychology:
    • Theory Theory (Murphy, 2002)
    • FUSS (Vigliocco Vinson et al)
semantic concept descriptions pustejovsky 1991 1995
SEMANTIC CONCEPT DESCRIPTIONSPUSTEJOVSKY (1991, 1995)
  • Lexical entries have a QUALIA STRUCTURE consisting of four ‘roles’
    • FORMAL role: what type of object it is (shape, color, ….)
    • CONSTITUTIVE role: what it consists of (parts, stuff, etc.)
      • E.g., for books, chapters, index, paper ….
    • TELIC role: what is the purpose of the object (e.g., for books, READING)
    • AGENTIVE role: how the object was created (e.g., for books, WRITING)
beyond bundles of attributes description logics theory theory
BEYOND BUNDLES OF ATTRIBUTES: DESCRIPTION LOGICS, THEORY THEORY
  • We know much more about concepts than the fact that they have certain attributes:
    • We know that cars have 4 wheels whereas bicycles have 2
    • We don’t just know that people have heads, bodies and legs, but that heads are attached in certain positions whereas legs are attached in other ones
    • Facts of this type can be expressed even in the simplest concept description languages, those of description logics
beyond simple relations description logics
BEYOND SIMPLE RELATIONS: DESCRIPTION LOGICS

Bear  (and Animal ( 4 Paw) …)

Strawberry  (and Fruit (fills Color red) … )

Female  (and Human ( Male))

word sense discrimination
WORD SENSE DISCRIMINATION
  • The senses of palm in WordNet
    • the inner surface of the hand from the wrist to the base of the fingers
    • a linear unit based on the length or width of the human hand
    • any plant of the family Palmae having an unbranched trunk crowned by large pinnate or palmate leaves
    • an award for winning a championship or commemorating some other event
concept acquisition meets relation extraction
CONCEPT ACQUISITION MEETS RELATION EXTRACTION
  • We developed methods to identify SEMANTIC properties of concepts (`Deep lexical relations’)
    • ATTRIBUTES and their VALUES
      • Almuhareb & Poesio 2004, 2005
    • Extracting QUALIA
      • Poesio & Almuhareb 2005
    • Letting relations emerge from the data: STRUDEL
      • Baroni et al, Cognitive Science to appear
    • Extracting Wu & Barsalou –style relations
      • Poesio Barbu Giuliano & Romano, 2008

We showed that for a variety of tasks such conceptual descriptions are ‘better’ than word-based or grammatical function-based descriptions

almuhareb poesio 2005 using a parser
ALMUHAREB & POESIO 2005: USING A PARSER

LOOKING ONLY FOR (POTENTIAL) ATTRIBUTES AND THEIR VALUES BETTER THAN USING ALL GRS

EVEN IF ATTRIBUTES OBTAINED USING TEXT PATTERNS (“THE X OF THE Y” )

supervised extraction of concept descriptions
SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS
  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)
  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)
supervised extraction of concept descriptions21
SUPERVISED EXTRACTION OF CONCEPT DESCRIPTIONS
  • Using a theory of attributes merging ideas from Pustejovsky and Guarino (Poesio and Almuhareb, 2005)
  • Using Wu and Barsalou’s theory of attributes (Poesio Barbu Romano & Giuliano, 2008)
the classification scheme for attributes of poesio almuhareb 2005
THE CLASSIFICATION SCHEME FOR ATTRIBUTES OF POESIO & ALMUHAREB 2005
  • PART
    • (cfr. Guarino’s non-relational attributes, Pustejovsky’s constitutive roles)
  • RELATED OBJECT
    • Non-relational attributes other than parts, relational roles
  • QUALITY
    • Guarino’s qualities, Pustejovsky’s formal roles
  • ACTIVITY
    • Pustejosvky’s telic and agentive roles
  • RELATED AGENT
  • NOT AN ATTRIBUTE (= everything else)
a supervised feature classifier
A SUPERVISED FEATURE CLASSIFIER
  • We developed a supervised feature classifier that relies on 4 types of information
    • Morphological info (Dixon, 1991)
    • Question patterns
    • Features of features
    • Feature use
      • Some nouns used more commonly as features than as concepts: i.e., “the F of the C is” more frequent than “the * of the F is”
  • (These last four methods all rely on info extracted from the Web)
the experiment
THE EXPERIMENT
  • We created a BALANCED DATASET
    • ~ 400 concepts
    • representing all 21 WordNet classes, including both ABSTRACT and CONCRETE concepts
    • balanced as to ambiguity and frequency
  • We collected from the Web 20,000 candidate features of these concepts using patterns
  • We hand-classified 1,155 candidate features
  • We used these data to train
    • A binary classifier (feature / non feature)
    • A 5-way classifier
outline25
OUTLINE
  • Vector space representations
  • An example of `Semantic-based’ vector space model
  • Evaluating such models
  • Attribute extraction and Feature norms
  • Category distinctions and Brain data
evaluation
EVALUATION
  • Qualitative:
    • Visual inspection
    • Ask subjects to assess correctness of the classification of the attributes
  • Quantitative:
    • Use conceptual descriptions for CLUSTERING (CATEGORIZATION)
quantitative evaluation
QUANTITATIVE EVALUATION
  • ATTRIBUTES
    • PROBLEM: can’t compare against WordNet
    • Precision / recall against hand-annotated datasets
    • Human judges (ourselves):
      • We used the classifiers to classify the top 20 features of 21 randomly chosen concepts
      • We separately evaluated the results
  • CATEGORIES:
    • Clustering of the balanced dataset
    • PROBLEM: The WordNet category structure is highly subjective
limits of this type of evaluation
LIMITS OF THIS TYPE OF EVALUATION
  • No way of telling how complete / accurate are our concept descriptions
    • Both in terms of relations and in terms of their relative importance
  • No way of telling whether the category distinctions we get from WordNet are empirically founded
beyond judges evaluation against wordnet
BEYOND JUDGES / EVALUATION AGAINST WORDNET
  • Task-based evaluation
  • Evidence from other areas of cognitive science
  • (ESSLLI 2008 Workshop - Baroni / Evert / Lenci )
task based black box evaluation
TASK-BASED (BLACK-BOX) EVALUATION
  • Tasks requiring lexical knowledge:
    • Lexical tests:
      • TOEFL test (Rapp 2001, Turney 2005)
    • NLP tasks:
      • Eg, anaphora resolution (Poesio et al 2004)
    • Actual applications
      • E.g., language models (Mitchell & Lapata ACL 2009, Lapata invited talk)
evidence from other areas of cognitive science
EVIDENCE FROM OTHER AREAS OF COGNITIVE SCIENCE
  • Attributes: evidence from psychology
    • Association lists (priming)
      • E.g., use results of association tests to evaluate proximity (Lund et al, 1995; Pado and Lapata, 2008)
      • Comparison against feature norms: Schulte im Walde, 2008)
    • Feature norms
  • Category distinctions: evidence from neural science
outline40
OUTLINE
  • Vector space representations
  • An example of `Semantic-based’ vector space model
  • How to evaluate such models
  • Attribute extraction and Feature norms
  • Category distinctions and Brain data
feature based representations in psychology
FEATURE-BASED REPRESENTATIONS IN PSYCHOLOGY
  • Feature-based concept representations assumed by many cognitive psychology theories (Smith and Medin, 1981, McRae et al, 1997)
  • Underpin development of prototype theory (Rosch et al)
  • Used, e.g., to account for semantic priming (McRae et al, 1997; Plaut, 1995)
  • Underlie much work on category-specific defects (Warrington and Shallice, 1984; Caramazza and Shelton, 1998; Tyler et al, 2000; Vinson and Vigliocco, 2004)
feature norms
FEATURE NORMS
  • Subjects produce lists of features for a concept
  • Weighed by number of subjects that produce them
  • Several existing (Rosch and Mervis, Garrard et al, McRae et al, Vinson and Vigliocco)
  • Substantial differences in collection methodology and results
comparing corpus features with feature norms almuhareb et al 2005 poesio et al 2007
COMPARING CORPUS FEATURES WITH FEATURE NORMS (Almuhareb et al 2005, Poesio et al 2007)
  • 35 concepts in common between the Almuhareb & Poesio dataset and the dataset produced by Vinson and Vigliocco (2002, 2003)
    • ANIMALS: bear, camel, cat, cow, dog, elephant, horse, lion, mouse, sheep, tiger, zebra
    • FRUIT: apple, banana, cherry, grape, lemon, orange, peach, pear, pineapple, strawberry, watermelon
    • VEHICLE: airplane, bicycle, boat, car, helicopter, motorcycle, ship, truck, van
  • We compared the features we obtained for these concepts with the speaker-generated features collected by Vinson and Vigliocco
results
RESULTS
  • Best recall: ~ 52% (using all attributes and values)
  • Best precision: ~ 19%
  • But: high correlation (ro=.777) between the distances between concept representations obtained from corpora, and the distances between the representations for the same concepts obtained from subjects (using the cosine as a measure of similarity)
discussion
DISCUSSION
  • Substantial differences in features and overlap, but correlation similar
  • Problems:
    • Each feature norm slightly different
    • They have been normalized by hand: LOUD, NOISY, NOISE all mapped to LOUD
problem differences between feature norms
Problem: differences between feature norms
  • motorcycle
    • Vinson & Vigliocco:
      • wheel, motor, loud, vehicle, wheel, fast, handle, ride, transport, bike, human, danger, noise, seat, brake, drive, fun, gas, machine, object, open, small, travel, wind
    • Garrard et al:
      • vehicle, wheel, fast, handlebar, light, seat, make a noise, tank, metal, unstable, tyre, coloured, sidecar, indicator, pannier, pedal, speedometer, manoeuvrable, race, brakes, stop, move, engine, petrol, economical, gears
    • McRae et al:
      • wheels, 2_wheels, dangerous, engine, fast, helmets, Harley_Davidson, loud, 1_or_2_people, vehicle, leather, transportation, 2_people, fun, Hell\'s_Angels, gasoline
    • Mutual correlation of ranks ranges from 0.4 to 0.7
discussion49
DISCUSSION
  • Preliminary conclusions: need to collect new feature norms for CL
    • E.g., use similar techniques to collect attributes for WordNet
    • See Kremer & Baroni 2008
  • For more work on using feature norms for conceptual acquisition, see
    • Schulte im Walde 2008
    • Baroni et al to appear
  • For the correlation between feature norms and information in WordNet (meronymy, isa, plus info from glosses): Barbu & Poesio GWC 2008
outline50
OUTLINE
  • Vector space representations
  • An example of `Semantic-based’ vector space model
  • How to evaluate such models
  • Attribute extraction and Feature norms
  • Category distinctions and brain data
using brain data to identify category distinctions
USING BRAIN DATA TO IDENTIFY CATEGORY DISTINCTIONS
  • Studies of brain-damaged patients have been shown to provide useful insights in the organization of conceptual knowledge in the brain
    • Warrington and Shallice 1984, Caramazza & Shilton 1998
  • fMRI has been used to identify these distinctions in healthy patients as well
    • E.g., Martin & Chao
  • See, e.g., Capitani et al 2003 for a survey
corpus data and brain data
CORPUS DATA AND BRAIN DATA
  • Can brain data (from healthy patients) be used to get an ‘objective’ picture of categorical distinctions in the brain?
  • Can our findings be useful to understand better the neurological results?
  • Ongoing project: using EEG and fMRI to identify such distinctions
eeg spectral analysis of concepts
EEG Spectral Analysis of Concepts
  • Participants presented with aural or visual concept stimuli
  • EEG apparatus records electrical activity on the scalp
  • Waveforms can be reduced to frequency components
eeg pros and cons
EEG pros and cons
  • Pros:
    • Lighter
    • Cheaper
    • Better temporal resolution (ms)
  • Cons:
    • Coarser spatial resolution (cm)
    • Noisy (e.g., very sensitive to skull depth)
a category distinction experiment with eeg
A CATEGORY DISTINCTION EXPERIMENT WITH EEG
  • Murphy Poesio Bovolo DalPonte & Bruzzone, Cogsci 2008
  • Seven Italian native speakers
  • Image stimuli only:
    • 30 tools
    • 30 animals
  • Each stimulus presented six times
  • Optimal time / frequency window identified automatically
    • 100-370ms
    • 3-17 Hz
data analysis
Data analysis
  • Preprocessing
    • Artefact removal
  • Feature extraction
    • CSSD: a form of supervised component analysis
  • Classification
    • Using SVMs
data analysis62

Classification System Schematic

64 channels

preprocessed data

X channels

filtered data*

“Tool” component

Feature vector

Filter by Time, Freq and Eelectr.

CSSD Decomposition

Vector Transform

var(“tool”), var(“animal”)

SupVec Machine

Answer

?

“Animal” component

Data analysis
representation of categories in cssd spaces
Representation of categories in CSSD spaces
  • component analysis identifies 2-dimensional spaces
  • Analysis of these spaces may provide useful data against which to compare our corpus models
brain data and corpus data
BRAIN DATA AND CORPUS DATA
  • What is the relation between the conceptual spaces induced from corpora with the conceptual spaces elicited using EEG?
predicting brain fmri activation using concept descriptions
PREDICTING BRAIN (FMRI) ACTIVATION USING CONCEPT DESCRIPTIONS
  • T. Mitchell, S. Shinkareva, A. Carlson, K. Chang, V. Malave, R. Mason and M. Just. 2008. Predicting human brain activity associated with the meanings of nouns. Science320, 1191–1195
mitchell et al 2008 methods
MITCHELL ET AL 2008: METHODS
  • Record fMRI activation for 60 nominal concepts
    • And extract 200 ‘best’ features, or VOXELs
  • Build conceptual descriptions for these concepts from corpora (the Web)
    • 25 features for each concept
    • 25 verbs expressing typical properties of living things / tools
    • Collect strength of association between these features and each concept
  • Learn association between each voxel and the 25 verbal features using 58 concepts
  • Use learned model to predict activation of 2 held-out data (compare using Euclidean distance)
    • Accuracy: 77%
our experiments
OUR EXPERIMENTS
  • Replicate the Mitchell et al study using EEG data instead of fMRI
    • Different feature selection mechanisms
  • Compare different methods for building concept descriptions
    • In addition to hand-picked, also a variety of standard corpus models
  • For Italian
  • B. Murphy, M. Baroni, M. Poesio, EEG responds to conceptual stimuli and corpus semantics, EMNLP 2009
recap
RECAP
  • We need to relate the evidence from corpora with evidence about concepts coming from empirical work in the neuroscience and psychology
  • Feature norms databases could be used to evaluate attribute extraction
    • But: need to find better ways of collecting them
  • Brain data may give us information about the ‘real’ conceptual categories
    • Results still preliminary
collaborators
COLLABORATORS

MARCO BARONI(Trento)

BRIAN MURPHY(Trento)

ABDULRAHMAN ALMUHAREB(Essex PhD 2006 Now at KACST, Saudi Arabia)

EDUARD BARBU(Trento PhD forthc)

HEBA LAKANY(formerly Essex, now Strathclyde)

thanks
THANKS

To the audience …..

And to Galja, Ruslan & the other organizers for yet another splendid RANLP!

ad