kth tmh stockholm loredana@speech kth se n.
Skip this Video
Loading SlideShow in 5 Seconds..
KTH-TMH Stockholm loredana@speech.kth.se PowerPoint Presentation
Download Presentation
KTH-TMH Stockholm loredana@speech.kth.se

Loading in 2 Seconds...

play fullscreen
1 / 27

KTH-TMH Stockholm loredana@speech.kth.se - PowerPoint PPT Presentation

  • Uploaded on

Loredana Cerrato. Looking for feedback expressions in speech corpora. KTH-TMH Stockholm loredana@speech.kth.se. General idea.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'KTH-TMH Stockholm loredana@speech.kth.se' - lysa

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
kth tmh stockholm loredana@speech kth se

Loredana Cerrato

Looking for feedback expressions

in speech corpora

KTH-TMH Stockholm


general idea
General idea

Study of the distribution, typology, use and function of verbal and non-verbal feedback expressions in human communication and in human machine communication.


Try to model human behaviour and try to reproduce it in speech technology application

definition of feedback in human communcation
Definition of feedback in human communcation

Allwood J. et al. 1992, On the semantics and pragmatics of linguistic feedback

“Control strategies of the channel of communication and of the informative flux, used by participant to a conversation to support the interaction”.

”Feedback mechanism is one of the most important cohesion devices in human conversation”

”Feedback consists of a set of verbal and bodily means used, in a minimal intrusive way, both to elicit and give information about the basic communicative functions: contact, perception, comprehension, attitudinal reactions”

general ways of expressing verbal feedback
General ways of expressing verbal feedback
  • short verbal expressions: yes, no, ok together with some prosodical and phonological phenomena (like varying pitch contour and vocalic lengthening)
  • short non lexical items: ah, ah-ah, mm, mh-mh
  • repetition:either the last word uttered by the interlocutor or by repeating the core words of the last sentence
  • reformulation: of the meaning of the received message.
general ways of expressing non verbal feedback
General ways of expressing non-verbal feedback
  • Facial expressions: smile, eyebrow rising, gaze.
  • Head movements: nods, shakes, tilts ..
  • Gestures: hand, shoulder movements
  • Postures: trunk position.

-Combination of verbal and non verbal expressions

Looking for verbal feedback expressions in a corpus of Italian spoken dialogues (human-human communication)


  • 12 spontaneous-elicited dialogues of ca. 12 minutes each
  • AVIP database: Archive of Spoken Italian Varieties

Napoli, Bari, Pisa

(Collected in 1998-1999, it containes 16 dialogues transcribed at different levels: ortographic, phonetic, phonemic, prosodic and DA)

  • Map task technique

Cerrato L. 1999


Map task technique


Only audio recordings

Task-oriented conversation

Giver: gives instruction

Follower: receives the instruction to accomplish the task



park? What park?

I don´t have any parks

Go straight

till the lake, then at

the park turn left.

the auditive channel of communication is maximised

no visual channel, therefore feedback expressions are produced in a great number.

  • Identification and labeling of feedback
  • Distributional analysis

3) Acoustic phonetic analysis:

    • Duration and Pitch contour analysis

With the aim of detecting acoustic cues related to the function carried out by feedback expressions:

  • show attention/continuation of contact
  • aknowledge reception
  • agreement, acceptance
  • refusal
  • require clarifications
  • require further explanation
  • require further indications
  • show reception and acquisition of knowledge
distribution of feedback expressions
Distribution of feedback expressions
  • 1/3 of the turns consist of feedback expressionsIf a dialogue counts c.a 260 turns,

90 are feedback expressions

  • Feedback is expressed in a non-intrusive way
  • Distinction between giving and requesting feedback
  • Regional variants

A contradiction: if feedback is expressed in a ”non-intrusive way” can it be counted as a turn of its own?

intentional f0 variation can modify the meaning of the feedback expression
Intentional F0 variation can modify the meaning of the feedback expression
  • “Ah”: acquisition of knowledge
    • Flat F0: the acquired knowledge satisfies a previous need of the listener
    • Rising-falling F0: the information received doesn’t satisfy a need, but disappoints the listener
Looking for verbal feedback expressions in a corpus of Swedish spoken dialogues (human-human communication)

Material: 4 real dialogues between

1 travel agent and 4 customers

recorded in a travel agency in Gothenburg

Göteborg Spoken Language Corpus.

Cerrato, L. 2002a

  • Identification and labeling of feedback
  • Distributional analysis
  • Analysis of its function:
    • show continuation of contact and perception
    • acknowledge comprehension
    • express a point of view (attitudinal reactions)

4) Acoustic phonetic analysis:

    • Duration and Pitch contour analysis
results customers agent distribution for each dialogue
Results:Customers-agent distribution for each dialogue
  • The number of feedback expressions of the customers reaches more than 50% in all the dialogues.
feedback functions
Feedback functions



Continuation of contact

Acknowledgement of information

Expression of a point of view

results of the acoustic analysis
Results of the acoustic analysis

·short duration and flat pitch contour

  • active listening attitude
  • not intend to take the turn;

·longer duration and rising pitch contour

  • active listening attitude intention to take the turn


290 ms


550 ms

conclusions on feedback
Conclusions on feedback
  • It is possible to categorize verbal feedback expressions according to their typology and to the meaning they convey.
  • Intentional variation of F0 seems to play an important role for assigning a particular meaning to the expressions used.
  • Question and hypothesis:

Is it possible to categorize visual feedback in the same way?

the study of visual feedback in collaboration with jens allwod c
The study of ”visual feedback”in collaboration with Jens Allwod & c.
  • What is the relation between co-occurring visual and verbal feedback expressions?
    • Do visual expressions emphasise the meaning of the verbal expression?
    • Or do they convey different shades of meaning?
  • When visual feedback occur independently of verbal feedback expressions what is its function and meaning?

Need for a multimodal database

feedback expressions are produced along different modalities
Feedback expressions are produced along different modalities
  • Verbal: words and sentences
  • Prosodic: speech rhythm, pauses, intensity, stress, intonation
  • Gestural: hand and arm movements
  • Facial: head and eye movement, gaze, smile and other facial expressions
  • Bodily:posture, trunk and leg movements
feedback coding in a multimodal corpus
Feedback coding in a ”multimodal corpus”
  • Tool: Multitool(Leif Grönqvist, Magnus Gunnarsson)
  • Corpus: GSLC- Travel agency activity
  • Annotation: Multitier annotation of feedback expressions

Text: transcription (MSO)

Function: reports the function of the feedback expression and the function of the utterance that precede the feedback expression

Gesture: reports the “non-verbal feedback expressions”, analysed in terms of movements.

Gaze: [should] report the direction of the speakers’ gaze

feedback in human machine interactions
System’s feedback

Process feedback

”I am active” ”I am retrieving information”

Information feedback

”I have understood”:explicit, implicit feedback that indicates the reception and comprehension of the message

Explicit feedback:

User: I want to go to London

System: did you say you wanted to go to London?

Implicit feedback:

User: I want to go to London

System: At what time do you wish to arrive in London?

Users’ feedback

Explicit, Implicit

To show:







Feedback in human machine interactions
analysis of users feedback in human machine interactions
Analysis of users’ feedback in human-machine interactions


94% of users (of 33) gave feedback at least once

18% of utterances (of 1845) contained feedback expressions

Only in 6% of these cases, feedback appeared in a turn of its own (it occurred mostly at the beginning of longer sentences)


  • 18%of users(of 1206) gave feedback at least once
  • 6% of utterances (of 6876) contained feedback expressions
  • In 89% of these cases, feedback appeared in a turn of its own.

Results adapted from Bell L. & Gustafson J. 2000;



  • Augustwas an experimental spoken dialogue system able to give information about tourist sights and restaurantsin Stockholm, about KTH and the life and works of August Strindberg.
  • The August database counts more than 10.000 utterances produced by 2685 users, all visitors at the Stockholm Cultural Centre, where the dialogue system was displayed as part of the Cultural Capital of Europe '98 program.

Adapt is an experimental dialogue system in the domain of Stockholm’ s real-estate.

The Adapt database was collected in 1999 by means of the Wizard of Oz technique.

It includes a total of 50 dialogues produced by 33 users.

Comparison between feedback strategies in Human-human communication and in human-machine communication
  • Material:

4 dialogues from GCSL and 4 dialogues from ADAPT

  • Method:

Distributional, functional and acoustic analysis

  • Results:
  • Humans produce feedback also when interacting with machines, but in different ways.
  • The majority of feedback expressions in GSLC travel agency dialogues consist of short monosyllabic words produced often on their own in a non-intrusive way.
  • In the interaction with ADAPT this kind of short feedback were never produced, they rather appeared at the beginning of longer sentences.

Cerrato L. 2002b

measuring head movements related to feedback in collaboration with link ping university
Measuring head movements related to feedbackin collaboration with Linköping university

Quantitative measurement of the entity of head movement related to feedback expressions with the Qualisys MacReflex motion tracking system.

A pilot study:

  • recording of 4 elicited dialogues between humans recorded in controlled situation,
  • transcription and coding of the dialogues
  • identification of “typical visual” feedback
  • quantification of the movements
  • implementation of the results in talking heads to make them look more natural.
main references
Main References
  • Allwod J ., 1999, The Swedish Spoken Language Corpus at Göteborg University. In Proceedings of Fonetik 99, Gothenburg Papers in Theoretical Linguistics 81.
  • Allwod J., Nivre J., Ahlsén E. , 1992, On the Semantics and Pragmatics of Linguistic Feedback in Journal of Semantics 64.
  • Allwood J., Grönqvist L., Ahlsén E., Gunnarsson M. ,2002 Annotations and Tools for an Activity Based Spoken Language Corpus", Proc. of 2nd SIGdial Workshop on Discourse and Dialogue, Aalborg, Denmark
  • Bell L & Gustafson .,2000 Positive and negative user feedback in a spoken dialogue corpus, In Proc of ICSLP 2000, Beijing, China.
  • Cerrato L, 1999 Il feedback verbale in dialoghi elicitati con la tecnica del map task. In Atti del GFS 10 Napoli, Italia, 25-35
  • Cerrato L. 2002a, Some charactersitcs of feedback expressions in Swedish, in TMH.OPSR Vol.43 Fonetik 2002 p. 101-104
  • Cerrato L. 2002b, A comparison between feedback strategies in Human-Human and Human-machine communciation in proceedings of ICSLP-2002, Denver, Colorado USA, (in press)