МУЛЬТИМОДАЛЬНАЯ ЛИНГВИСТИКА Семинар

1. МУЛЬТИМОДАЛЬНАЯ ЛИНГВИСТИКА Семинар Дульзоновские чтения Томск, 2011 А.А.Кибрик (Институт языкознания РАН) aakibrik@gmail.com

2. The mainstream linguistic approach • Language consists of hierarchically organized segmental units, such as phonemes, morphemes, words, phrases, and sentences • Linguistic form is thus equated with verbal form • Search for “linguistic form” in Google: • The first result is: • “A meaningful unit of language, such as an affix, a word, a phrase, or a sentence.” (TheFreeDictionary.com) • «В своей совокупности языковые знаки образуют особого рода знаковую систему – язык. <…> Наиболее типичным языковым знаком является слово <…> Форма выражения любого словесного знака состоит из фонем» (Лингвистический энциклопедический словарь, с. 167)

3. However • Apart from sound, there are other channels of communication, in the first place through vision (body language - gesture, mimics, gaze, etc.) • There are prosodic, that is non-verbal aspects to sound • Imagine prosody-free talk • or, vice versa, talk behind a wall

4. Multimodality • In order to understand language and communication, all aspects of linguistic form must be taken into account • This is what is sometimes called the multimodal approach • Modality, or mode, refers to a distinct type of input • In particular, modality is a kind of stimulus associated with one the human senses, particularly hearing and sight • So the verbal component, prosody, and body language all count as modes or modalities • “Any use of language is inescapably multimodal” (Scollon 2006)

5. Goals of this talk • Emphasize the importance of prosody and visual aspects of communication in linguistic research • Show how prosody and visual communication interact with the verbal component, thus suggesting not only the multimodal, but also the cross-modal approach • Propose that linguistics cannot progress without taking multimodality seriously into account

6. Are these goals relevant and important? • After all, linguists and other scholars have already been pursuing these issues for many decades, and the respective research traditions are quite rich • But: • First, prosody and visual communication are marginalized in linguistics, they are located in certain “pockets” of the overall linguistic panorama and are tolerated by the mainstream as “paralinguistics” • Those focusing on these information channels often treat them as a “thing in itself”, without integration with the verbal component

7. Plan of talk • I. Prosody • II. Gestures • III. Relative contribution of three information channels • IV. Signed languages • V. Wider context

8. I. PROSODY • Prosodic components • pausing • accents • pitch • tempo (of various scope) • registers • degrees of reduction • glottal features • loudness • ................ • «Рост интереса к просодии связан <…> с новыми семантическим задачами (описание непропозициональной семантики <...>)» (Кодзасов 1996: 85) • Prosody is responsible for discourse segmentation into Elementary Discourse Units (EDUs), identified on the basis of several prosodic components and strongly correlated with clauses

9. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3) (Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2) как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0) ’и-иh ...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero ...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW ...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0) ’i-iH ...(0.7) čerez /nego ..(0.3) kak-to \brevnokakoe-to, tipa \mosta. An example of prosodically oriented discourse transcription

10. Night Dream Stories • Corpus of spoken Russian stories • Speakers: children and adolescents • Subject matter: retelling of night dreams • Discourse type: monologic narrative (personal stories) • Joint study with Vera Podlesskaya and a group of our graduate students • Kibrik and Podlesskaya eds. 2009

11. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3) (Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2) как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0) ’и-иh ...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero ...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW ...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0) ’i-iH ...(0.7) čerez /nego ..(0.3) kak-to \brevnokakoe-to, tipa \mosta. Segmentation (EDUs)

12. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3)(Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2)как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0) ’и-иh...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0)’i-iH...(0.7)čerez /nego..(0.3)kak-to \brevnokakoe-to, tipa \mosta. Pauses

13. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3) (Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2) как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0) ’и-иh ...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero ...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW ...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0) ’i-iH ...(0.7) čerez/nego ..(0.3) kak-to\brevnokakoe-to, tipa\mosta. Pitch accents

14. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3) (Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2) как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0) ’и-иh ...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero ...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW ...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0) ’i-iH ...(0.7) čerez /nego ..(0.3) kak-to \brevnokakoe-to, tipa \mosta. Tempo: wide and narrow scope

15. ....(1.5) /\Озеро ...(0.5)какое-то, Lake some ..(0.3) (Или /\речка, Either river или /\озеро, or lake но по-моему \озеро, but I guess lake потому что’..(0.2) как-то-оwbecause somehow...(0.6)\маленькоетакое, small such \небольшое.) minor ....(1.0)’и-иh ...(0.7)через/него and across it..(0.3)как-то\бревнокакое-то, somehow log some типа\моста. like bridge ....(1.5) /\Ozero ...(0.5) kakoe-to, ..(0.3) (Ili/\rečka, ili/\ozero, no po-moemu\ozero, potomučto’..(0.2) kak-to-oW ...(0.6) \malen’koetakoe, \nebol’šoe.) ....(1.0) ’i-iH ...(0.7) čerez /nego ..(0.3) kak-to \brevnokakoe-to, tipa \mosta. Other prosodic phenomena

16. Prosody and sentence • Does spoken language consist of sentences? • Sheer facts: • Spoken language is the primary form of language • Spoken language does not contain periods, question marks and other explicit signals of sentence boundaries • Research question: • Is sentence, as a theoretical construct, as identifiable and as basic for the primary form of language as it is (or as it is thought to be) for written language?

17. Sentence in spoken language • Position 1: sentence is a universal and basic unit of language • Assumption typically held by not only by linguists but also by other cognitive scientists • But sentence is very far from being obvious in spoken language • Position 2: avoidance of the issue, typical of discourse-oriented linguists • If so, how could sentences become so much entrenched in written language?

18. Phase (фаза) • Term by Sandro V. Kodzasov • Alternative term by J. DuBois et al. 1992: transitional continuity • Discourse semantic category: ‘end’ vs. ‘non-end’ (=expectation of a forthcoming end) • Hierarchical nature of phase • End of tentative sentence – falling tonal accent • Non-end – rising tonal accent

19. A canonical example of the transitional continuity distinction z57:15-16 • ..(0.4) /\Мы-ы’ ..(0.4) \как бы за них /взя-ались, ..(0.4) /\My-y’ ..(0.4) \kakbyzanix /vzja-alis’, We sort of at them got.hold • ...(0.5) и-и ввь= || ..(0.2) полетели \вве-ерх. • ...(0.5) i-i vv’= || ..(0.2) poleteli \vve-erx. • and flew upward • Rising (“comma”) • Non-end • Falling (“period”) • End • If things were that easy, sentence would be uncontroversial

20. ....(1.5) /\Озеро ...(0.5)какое-то, ..(0.3) (Или /\речка, или /\озеро, но по-моему \озеро, потому что’..(0.2) как-то-оw...(0.6)\маленькоетакое, \небольшое.) ....(1.0) ’и-иh ...(0.7)через/него..(0.3)как-то\бревнокакое-то, типа\моста. ....(1.5) /\Ozero ...(0.5) kakoe-to, Lake some ..(0.3) (Ili/\rečka, Either river ili/\ozero, or lake no po-moemu\ozero, but I guess lake potomučto’..(0.2) kak-to-oWbecause somehow...(0.6) \malen’koe takoe, small such \nebol’šoe.) minor ....(1.0) ’i-iH ...(0.7) čerez /negoand across it ..(0.3) kak-to\brevno kakoe-to,somehow log some tipa\mosta.like bridge Non-canonical situation: Non-end with a falling tonal accent

21. The problem of two kinds of falling • The existence of non-final falling calls relevance of sentence into question • However, the distinction between two kinds of falling is very systematic • The two kinds of falling: • are prosodically distinct • have distinct discourse functions

22. Prosodic criteria of the final vs. non-final falling distinction • Target frequency band • Post-accent behavior • Pausing pattern • Reset vs. latching • Steepness of falling • Interval of falling

23. Target frequency band • Final falling (“period”): targets at the bottom of the speaker’s F0 range • Non-final falling (“falling comma”): targets at level several dozen Hz (several semitones) higher

24. F0 graph for the “lake” example 12 10 12 8 5 \ozero, \malen’koe \nebol’ \brevno kakoe \mosta. takoe, šoe. -to,

25. Non-final falling (210 Гц),final falling (170 Гц),rising, post-rising falling Z54: 4-5 170 Hz 210 Hz

26. Post-accent behavior • Final falling (“period”): steady falling on the post-accent syllables • Non-final falling (“comma”): lack of falling on post-accent syllables, often rise of tone (V-curve)

27. V-curvez26 260 Hz 240 Hz 235 Hz

28. The final vs. non-final falling distinction • A speaker’s prosodic pattern must be identified • On its basis the difference between final and non-final falling distinction can be identified with a high degree of robustness

29. Contexts of non-final falling • Anticipatory mirror-image adaptation • Inset • Stepwise falling

30. Anticipatory mirror-image adaptation • ....(1.8)Когдая\услышала, Kogda ja \uslyšala,when I heard • ...(0.5)что-о/бомбагремит, čto-o /bomba gremit, that bomb growls

31. Inset • /Входитэто ...(0.5)/\ма-аль↑чик,/Vxoditèto ...(0.5) /\ma-al’↑čik, enters here boy • ’ ’ ..(0.1)/\нук\другому, ’ ’ ..(0.1) /\nu k \drugomu, well to another • ..(0.1) и\говорит: ..(0.1) i \govorit: and says

32. ....(1.5) /\Озеро ...(0.5)какое-то, ..(0.3) (Или /\речка, или /\озеро, но по-моему \озеро, потому что’..(0.2) как-то-оw...(0.6)\маленькоетакое, \небольшое.) ....(1.5) /\Ozero ...(0.5) kakoe-to, Lake some ..(0.3) (Ili/\rečka, Either river ili/\ozero, or lake no po-moemu\ozero, but I guess lake potomučto’..(0.2) kak-to-oWbecause somehow...(0.6) \malen’koe takoe, small such \nebol’šoe.) minor Stepwise falling 210 Hz 190 Hz 160 Hz

33. Representation of EDU continuity types in corpus

34. The status of sentence • In the speech of most speakers final falling is clearly distinct from non-final patterns • Final intonation, expressly distinct from non-final intonation (both rising and falling), makes the notion of sentence valid for spoken discourse • Speakers “know” when they complete a sentence and when they do not • Apparently, spoken sentences are the prototype of written sentences

35. However • Identification of sentences is possible only on the basis of a complex analytic procedure • It is dependent on prior understanding of a speaker’s prosodic “portrait” • There are prototypes of final and non-final fallings, but there are intermediate instances, therefore sentencehood may be a matter of degree • Unlike EDUs, sentences are highly variable • Speakers with short sentences • Speakers with long sentences equaling stories • Clause chaining • A significant tune-up is necessary to apply the procedure to a different discourse type or a different language

36. Conclusions on prosody and sentence • Sentence is an intermediate hierarchical grouping between an EDU (roughly, clause) and whole discourse • Sentence is an elusive, complex, non-elementary unit of spoken language • These conclusions, possible only due to prosodic analysis, are of prime importance for linguistic theory • The notion of sentence, so salient in theories restricted to the verbal component alone, can only be evaluated relying on prosodic evidence

37. Other languages? • Upper Kuskokwim Athabaskan • Bobby Esai, Sr.

38. Excerpt from a story a. (1.6) hwndine ŒiÈ chu suddenly with Ptcl b. (2.2) sighwdlaŒ todoltsitÈ' ts'eŒ my.sled it.broke.through.ice and c. (5.5) sileka ch'ildon' nich'i toghedak Œedinh my.dogs some too they.fell.in.water though d. (0.9) ch'ildon' chuŒda some though e. (0.2) tinh k'its' == ice on f. (0.9) tinh k'its' Œohighet'a ts'eŒ ice on they.are.there and ‘Suddenly, my sled broke through the ice, and some of my dogs also fell into the water, while others remained on top of the ice, and <…>’

39. Tonal contours and EDUs a b c d e f

40. II. GESTURE • In the course of communication, it is not just that the speaker speaks and the addressee listens • In addition, the speaker displays, and the addressee observes • Gesture • Gaze • Mimics • Posture • Proxemics • Cultural symbolism • ..................... (see, for example, Крейдлин 2002, Бутовская 2004)

41. Gestures • Gestures are kinetic behaviors of arms and other limbs, capable of conveying meaning from speaker to addressee. • Among the various types of gestures (see e.g. McNeill 1992) pointing gestures are one of the most salient types.

42. Pointing • Понюхай эти!

43. Elements of a canonical pointing act

44. Phylogeny and ontogeny • Appear an exclusive property of humans (Tomasello et al. 2007) • Are a very ancient gesture type (Крейдлин 2007) • Appear at the end of the first year • Can participate in binary multimodal constructions “word + gesture”, such as open POINT (Butcher and Goldin-Meadow 2000)

45. Reference and pointing • Reference is a fundamental linguistic phenomenon, accounting for about every third word in running discourse • Studies of reference (deixis, anaphora, etc.) among the central concerns of modern linguistics • Pointing is the developmental source of reference

46. Pointing, deixis, and exophora • Deixis is the most widely recognized function of pointing • However, quite frequently pointing is associated with exophora, that is mention of perceptually activated referents (O'Neill 1996, Levy 2000: 219, Nikolaeva 2003) • Exophora is the ontological source of anaphora

47. Exophoric and anaphoric reference (from Nikolaeva 2003) • a. My s Anatoliem uže mnogo let očen’ rabótaem, <three intervening clauses> • e. on mnogo raz zavjázyval, ‘Anatolij and I have been working together for many years, <…> he was winding it up (drinking) many times’

48. Pointing and prosody • Pointing and accentuation are analogous phenomena, both associated with making an item salient • Nikolaeva (p.c.): pointing typically cooccurs with accent • Levy (2000): energy expenditure

49. Substitution: Referent vs. demonstratum • Reference to non-specific items: Vot počemu my i obraščàemsja poroj k psixologam. ‘This is why we address psychologists now and then’ • This phenomenon is known as deferred ostension, analogic deixis, ostensive metonymy, etc. • In substitution, reference does not have to be non-specific He got a big scar here (pointing to one’s cheek) (Levelt 1989)

50. Virtual pointing • Pointing to imaginary targets • cf. Buehler’s Deixis am Phantasma, McNeill’s abstract pointing