Experiments on Building Language Resources for Multi-Modal Dialogue Systems

1. User Scenarios • useful in the context of lacking real human-machine interactions; • designed to obtain homogeneous linguistic coverage, for all the languages: • several styles (or registers - familiar, elaborated); • specific phrases (politeness phrases, time intervals - "from the sixties"); • various syntactic components (passive constructions, relative clauses, questions and ellipses); • dates or names • developers worked independently • building exhaustive user scenarios • Context-free grammar • covers the linguistic phenomena from the user scenario, for every language • Technical vocabulary • covers the linguistic phenomena from the user scenario, for every language • Adapting parser’s resources • TAG widely accepted formalism for syntactic parsing • XML standard for grammars (TAGML) and for semantic representations (MMIL) • existing resources for English, French for free texts • Speech recognizer’s language model • Statistical language model • 400 words vocabulary • Huge number of possible sentences • Training corpus: generated with context-free grammar. Two steps: • Bigram model using classes (ex: P(DECADES|the) • Uniform distribution of words into classes: P(90’s|DECADES) Developping the language resources L.Romary*, A.Todirascu**, D.Langlois* *LORIA, Nancy, France **Université de Technologie de Troyes, France Experiments on Building Language Resources for Multi-Modal Dialogue Systems • Goals • identification of a methodology for adapting linguistic resources for human-machine dialogue systems, without training corpora; • Multilinguality supported, uniform linguistic coverage of speech interpretation modules • case study: synchronising parser’s and SR interface for French for MIAMM project • The Multimedia Information Access Using Multiple Modalities (MIAMM) Project (2002-2003) • A prototype of a human-machine dialogue interface regrouping various interaction modalities: speech, haptics, graphics; • Case study: searching music into an existing database using all the modalities • Multilinguality supported: English, French , German • difficulties: multi-modal training corpora not available, Information flow: several models for the same language (one for speech, one for parsing : how to cover the same language),changing specifications during the project • Parser’s Experiments • iteration updating process after interaction with other modules • adding new lexical entries • local grammars • generated by a meta-grammar • preference for substitution • Domain-specific syntactic components (elliptic phrases, navigation verbs, noun groups) • mapping lexical entries to domain ontology • transforming derivation trees into MMIL representations • Speech Recognition’s Experiments • system: ESPERE (small vocabularies) • protocol: 88 sentences recorded, 4 speakers, 2 men and 2 females, no OOV, all sentences in language • several methods for estimating P(w|v) • Frequencies in training corpus: WER = 3.8, SD = 0.9 • Uniform probabilities for bigrams in training corpus: WER = 4.0, SD = 0.4 • True frequencies are not useful • All bigrams are possible (not null probabilities, back-off with ): WER >> 4.0 • Constraint bigrams from grammar are necessary • Training corpus (A) is adaptation corpus for a general newspaper one (B): • Linear combination  performance are less good than when A is used alone