The Biological Bases of Syntax-Semantics Interface in Natural
This presentation is the property of its rightful owner.
Sponsored Links
1 / 28

The Biological Bases of Syntax-Semantics Interface in Natural Languages: Cognitive Modeling and Empirical Evidence. PowerPoint PPT Presentation


  • 49 Views
  • Uploaded on
  • Presentation posted in: General

The Biological Bases of Syntax-Semantics Interface in Natural Languages: Cognitive Modeling and Empirical Evidence. Evguenia Malaia Ronnie B. Wilbur SLHS Department, Purdue University [email protected] event structure. WORLD-to-LANGUAGE. events in real world 

Download Presentation

The Biological Bases of Syntax-Semantics Interface in Natural Languages: Cognitive Modeling and Empirical Evidence.

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


The biological bases of syntax semantics interface in natural languages cognitive modeling and empirical evidence

The Biological Bases of Syntax-Semantics Interface in NaturalLanguages: Cognitive Modeling and Empirical Evidence.

Evguenia Malaia

Ronnie B. Wilbur

SLHS Department, Purdue University

[email protected]

event structure


World to language

WORLD-to-LANGUAGE

events in real world 

parsed out and expressed in sentences 

Each sentence centered around a predicate/verb

…and this is what we want to model


Parsing observable events

Parsing observable events:

Human languages parse and formulate observable events in a logically restricted fashion

(e.g. Son and Cole, 2008, Borer 1994, Ritter and Rosen 1998, Davis and Demidarche 2000, van Hout 2000, Hale and Keyser 1993, Van Valin 2007)

Vendler (1967)

four syntactically relevant semantic types of predicates:

  • homogenous // atelic States and Activities

    sleep (for an hour)

    walk (for an hour)

  • transitions // telic Achievements and Accomplishments

    die (in an hour)

    paint a portrait (in two weeks)


Model of event structure

Model of Event Structure

Atelic

Telic transition to end state

Pustejovsky 2001


Model of event structure1

Model of Event Structure

Ramchand (2008) - Divide events into

  • Initiation phraseInitP

  • Process phraseProcP

  • Result phraseResP

    event structure

    Participant involved:

  • Initiation phrase: Initiator

  • Process phrase:Undergoer

  • Result phrase:Resultee

    argument structure


Ramchand event structure model tree

Ramchand Event Structure Model Tree

initP (vP)

procP

Initiator

init

resP

Undergoer

proc

Resultee

res

(Rheme)

XP

- verbal morphology of individual languages can represent individuated elements of event structure…

- which allows use of a single verbal root in multiple event structures…

- to yield telic or atelic meanings.


Linguistic universals meet language processing

Linguistic universals meet language processing

Do we have empirical evidence that telicity affects the way human languages are processed?

Options:

complete analysis of all known languages of the world with respect to their event structure

evidence from psycholinguistic and neurolinguistic research


Linguistic universals meet language processing1

Linguistic universals meet language processing

Reaction times:

- O’Bryan (2003) - independent effects of telicity and transitivity

:word maze experiment with reduced relative clauses.

:reaction time advantage on the preposition “by” for sentences with telic verbs in the relative clause, as compared to those with atelic verbs;

:independent advantage for the second argument in sentences with obligatorily transitive verbs (both telic and atelic).

Friedmann, Taranto, Shapiro, and Swinney (2008)

priming effect for obligatorily intransitive telic verbs, but not for unergatives obligatorily intransitive atelics.

ERPs

- Malaia, Wilbur, Weber-Fox (in press) ERP evidence for telicity effects on syntactic processing in garden-path sentences. Brain and Language.

The baby nursed/ burped by the mother rolled over.

The customer shaved / cheated by the barber left no tip.

The prisoner taught/ halted by the agent tried to escape.


Erps on the rrc hp group

ERPs on the RRC: HP group

9


Erps on the rrc np group

ERPs on the RRC: NP group

10


Erps on the rrc hp and np groups

ERPs on the RRC: HP and NP groups

processing of syntactic information interacts with the previous semantic context.

(cf. Yamada & Neville, 2007; Osterhout, Holcomb, Swinney, 1994)

11


Frame alternations telic verbs

Frame alternations: telic verbs

awakened (transitive)

awakened (intransitive)

writer

actress

actress

New argument does not require re-assignment of thematic roles

12


Frame alternations atelic verbs

Frame alternations: atelicverbs

New argument requiresthematic role re-assignment

worshipped (intr.)

worshipped (transitive)

actress

writer

actress

13


Linguistic universals language processing unified account

Linguistic universals, language processing – unified account?

Where did

grammatically relevant semantic features

come from?

Research on Signed languages:

  • tied to the visual modality

  • in both production and perception

  • provide the missing link


Event structure in american sign language

Event structure in American Sign Language

Telic event signs

a. change of aperture b. orientation change c. setting change d. change of location

handshape change proximal/distal with contact

SEND HAPPEN POSTPONE HIT

Atelic event signs

(a) RUN [tracing: straight](b) PLAY (tracing + TM) (c) READ (tracing + TM)

Telic and atelic predicates in signed languages

15


Event structure and sls

Event Structure and SLs

We already know that telicity plays a role in sign morphology:

  • [delayed completive] aspect only applies to telic stems (Brentari 1998).

  • Durative and continuative aspects cannot apply to telic predicates (Wilbur 2005, in press).

  • Some mouth non-manuals are distributed according to predicate telicity type (Schalber 2004, 2006; Schalber & Grose 2006) in Austrian Sign Language and American Sign Language.

    :: production differences reflecting semantic distinction of event type


Event visibility hypothesis

Event Visibility Hypothesis

The Event Visibility Hypothesis (EVH) argues that the semantics of event structure (subevents) are visible in predicate sign formation.

Can see if an end state is intended by way sign movement comes to a stop:

- rapidly for end states

- regularly for no end state.


Motion analysis

Motion Analysis

ASL signs representing telic events appear to contain ‘perceptually significant rapid deceleration to a stop’.

Hypothesis:

Signs representing telic events will have steeper deceleration slopes than those representing atelic events, because it will provide an end-marking to indicate the final State.


Methodology

Methodology

Stimuli:

  • 29 telic and 21 atelic signs chosen, randomized and presented to native bilingual ASL signer.

    Conditions

  • Signs in isolation (done twice)

  • Signs in carrier phrase SIGN X AGAIN

  • Signs in medial sentence position IX3 X TODAY

  • Signs in final position TODAY IX3 X (NB: data analysis not yet finished).

  • Each sign produced 4 times by signer

    x 29 for telic = 116 cases for telics

    x 21 for atelic = 84 cases for atelics.

    Equipment

  • Gypsy 3.0 wired motion capture suit

  • Pair of 18-sensor Metamotion Cybergloves.

  • Six special motion capture ceiling mounted cameras.


Deceleration results

DecelerationResults

e.g. Mean telic slope is 1.46 times steeper than mean atelic slope in isolation

  • Sign movements have internal structure based on velocity changes

  • Signers are making production distinctions

Metric: the Slope from the velocity peak to the next velocity minimum, reflecting the deceleration to(ward) a stop.

Slope Atelic Telic Ratio Telic/Atelic

Isolation 1 * -.09 -.14 1.46

Isolation 2 * -.12 -.18 1.46

Carrier Phrase** -.12 -.23 1.97

Sentence 1 * -.14 -.23 1.62

Difference between atelic and telic slopes is significant at *p<.05 **p<.001


What is recipient doing for processing

What is recipient doing for processing?

ASL productionstudies - left hemispheric activation of Broca’s area (cf. Corina et al. 1999, Horowitz et al. 2003, Emmorey 2002, 2003, 2004).

ASL comprehension studies - bilateral activation in Broca’s area (Levänen et al, 2001, Neville et al., 1998, etc)

- also typical for audiovisual stimuli comprehension in spoken languages (cf. Capek et al., 2004).

Broca’s area is activated during both syntactic and semantic processing in spoken languages - integrates syntactic and semantic information during sentence comprehension (Hagoort, 2005).

:: spoken and signed languages appear to be highly correlated in the use of Broca’s area in the left hemisphere for integration of structural and lexical linguistic information.

21


Hypotheses on asl predicates

Hypotheses on ASL predicates

Telic predicates in ASL incorporate the end-state and internal argument

Atelic predicates do not  simpler  smaller load?

Hypothesis: ASL predicate signs with distinct types of event structure (telic vs. atelic) would elicit differentiated activation patterns in Broca’s area (BA 44/45) of left IFG.


Pilot fmri study of asl predicates

Pilot fMRI study of ASL predicates

Participants: 5 healthy adults - native ASL signers:

Subjects were presented with visual stimuli consisting of telic and atelic ASL signs in a block paradigm, with non-ASL gesture as a baseline condition.

23


Data acquisition and fixed effects analysis

Data acquisition and fixed effects analysis

Subtraction paradigm was used to compare differential activation during processing of telic vs. atelic predicates.

Fixed effects analysis of the pilot data from 5 subjects (p<.001, uncorrected for multiple comparisons) demonstrates activation clusters in Broca’s area (BA 44/45, cluster size 15 voxels), and V5/MT+, cluster size 5 voxels.

24


Interpreting the empirical data

Interpreting the empirical data

  • experiments demonstrate semantics/grammar interaction in spoken and signed languages

  • syntax of human languages cross-modally is grounded in what can be construed as biological perception.

    The way events are perceived and conceptualized is explicitly coded in syntax-semantics interface.

25


References

References

Bornkessel, I., & Schlesewsky, M. (2006). The Extended Argument Dependency Model: A Neurocognitive Approach to Sentence Comprehension Across Languages. Psychological Review, 113(4), 787-821.

Brentari, D. (1998). Prosodic Model of ASL. Cambridge: MIT Press.

Capek, C.M., Bavelier, D., Corina, D., Newman, A.J., Jezzard, P., & Neville, H.J., 2004. The cortical organization of audio-visual sentence comprehension: An fMRI study at 4 Tesla. Cognitive Brain Research, 20, 111-119.

Corina, D.P., San Jose-Robertson, L., Guillemin, A., High, J., & Braun, A. 2003. Language lateralization in a bimanual language. Journal of Cognitive Neuroscience, 15, 718-730.

Corina, D.P., McBurney, S.L., Dodrill, C., Hinshaw, K., Brinkley, J., & Ojemann, G. 1999. Functional roles of Broca’s area and supramarginal gyrus: Evigence from cortical stimulation mapping in a deaf signer. NeuroImage, 10, 570-581.

Emmorey, K. 2002. Language, cognition, and the brain: Insights from sign language research. Mahwah, NJ: Lawrence Erlbaum and Associates.

Emmorey, K., Grabowki, T., 2004. Neural organization for sign versus speech production. Journal of Cognitive Neuroscience Supplement, F69, 205.

Emmorey, K., Herzig, M., 2003. Categorical versus gradient properties of classifier constructions in ASL. In K. Emmorey (Ed)., Perspectivees on classifier constructions in sign languages. Mahwah, NJ: Lawrence Erlbaum and Associates.

Frazier, L., & Rayner, K. (1982). Making and correcting errors during sentence comprehension: eye movements in the analysis of structurally ambiguous sentences. Cognitive Psychology, 14, 178-210.

Frisch, S., & Schlesewsky, M. (2001). The N400 indicates problems of thematic hierarchising. Neuroreport, 12, 3391-3394.

Just, M. A., & Carpenter, P. A. (1992). A capacity theory of comprehension: Individual differences in working memory. Psychological Review(99), 122-149.

Just, M. A., Carpenter, P. A., Keller, T. A., Eddy, W.F., & Thulborn, K.R. (1996). Brain Activation Modulated by Sentence Comprehension. Science, 274(5284), 114-116.

Hagoort, P. (2005). On Broca, brain, and binding: a new framework. Trends in Cog. Sci. 9, 416–423.

Hopf, J., Bayer, J., Bader, M., & Meng, M. (1998). Event-Related Brain Potentials and Case Information in Syntactic Amgibuities. Journal of Cognitive Neuroscience, 10(2), 264-280.

Kaan, E., & Swaab, T.Y. (2003). Repair, revision, and complexity in syntactic analysis: an electrophysiological differentiation. Journal of Cognitive Neuroscience, 15(1), 98-110.

Kaan, E., Wijnen, F., & Swaab, T.Y. (2004). Gapping: Electrophysiological evidence for immediate processing of “missing” verbs in sentence comprehension. Brain and Language, 89, 584-592.

King, J., & Kutas, M. (1995). Who did what and when? Using word- and causal- level ERPs to monitor working memory usage in reading. Journal of Cognitive Neuroscience, 7(3), 376-395.

Levänen, S., Uutela, K., Salenius, S., & Hari, R. 2001. Cortical representation of sign lanugage: Comparison of deaf signers and hearing non-signers. Cerebral Cortex, 11, 506-512.

Neville, H., Bavelier, D., Corina, D., Rauschecker, J., Kami, A., Lalwani, A., et al. 1998. Cerebral organization for language in deaf and hearing subjects: Biological constraints and effects of experience. Proceedings of the National Academy of Science USA, 95, 922-929.

Malaia, E., Borneman, J., Wilbur, R.B. (2008) Analysis of ASL motion capture data towards identification of verb type. Symposium on semantics in systems for text processing, Venice, Italy, September 22-24.

Malaia, E., Wilbur, R., Weber-Fox, C. (accepted) Effect of telicity on syntax-semantics integration in sentence processing: ERP evidence. Brain and Language

O'Bryan, E. (2003). Event Structure in Language Comprehension. Unpublished manuscript, University of Arizona.

Osterhout, L., Holcomb, P. J., & Swinney, D. A. (1994). Brain potentials elicited by garden-path sentences: Evidence of the application of verb information during parsing. . Journal of Experiment Psychology: Learning, Memory, & Cognition, 20, 786-803.

Pustejovsky, J. (1991). The syntax of event structure. Cognition, 41(1-3): 47-81.

Ramchand, G. (2008). Verb Meaning and the Lexicon: A First Phase Syntax. Cambridge: Cambridge University Press.

Rathmann, C. (2005).Event Structure in ASL. Doctoral dissertation, University of Texas-Austin.

Sanz, M. (2000). Events and predication. A new approach to syntactic processing in English and Spanish. Amsterdam: John Benjamins.

Seegmiller, M., Ingraffea, K., & Townsend D. (2003). Role of telicity in sentence comprehension. Presentation at the “It’s About Time” workshop on Tense, Aspect, Modality, and Events. Michigan State University, Lansing.

Streb, J., Henninghausen, E., & Rösler, F. (2004). Different Anaphoric Experssions are Investigated by Event-Related Brain Potentials. Journal of Psycholinguistic Research, 33(3), 175-201.

Vendler, Z. (1967). Linguistics in Philosophy, Cornell University Press, New York.

Van Valin (2007) Some universals of verb semantics. In Linguistic Universals, Mairal, R., Gil, J. eds. Cambridge University Press.

Weber-Fox, C., & Neville, H. J. (2001). Sensitive periods differentiate processing of open- and closed-class words: An ERP study of bilinguals. Journal of Speech Language and Hearing Research, 44(6), 1338-1353.

Yamada, Y., Neville, H.J. (2007). An ERP study of syntactic processing in English and nonsense sentences. Brain Research, 1130(1), 167-180.

Wilbur, R. B. (2003). Representations of telicity in ASL. Chicago Linguistic Society 39 (1): 354-368.

26


Thank you

THANK YOU

We are grateful to Robin Shay, Gabriel Masters, Nicoletta Adamo-Villani, Greg Tamer, Javier Gonzalez-Castillo, and Purdue and Indianapolis sign language community for their ongoing support of Purdue Sign Language Lab research.

This work was supported by NSF Research in Disabilities Education grant, and by NIH grant DC00524 to R.B. Wilbur.

27


Ramchand s factorical typology of predicates

Ramchand’s factorical typology of predicates


  • Login