1 / 29

CLINT-CS

CLINT-CS. Dialogue II Verbmobil. Verbmobil. Verbmobil is a spoken dialogue system that provides phone users with simultaneous dialogue interpretation services for restricted topics. Recognises spoken input, translates it, and then utters the translation.

sauda
Download Presentation

CLINT-CS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. CLINT-CS Dialogue II Verbmobil CLINT-CS Verbmobil

  2. Verbmobil • Verbmobil is a spoken dialogue system that provides phone users with simultaneous dialogue interpretation services for restricted topics. • Recognises spoken input, translates it, and then utters the translation. • Three languages: German, English and Japanese CLINT-CS Verbmobil

  3. Challenges for S and L Technology Increasing difficulty CLINT-CS Verbmobil

  4. Grand Challenges • Not a push-to-talk system. Has to decide for itself when user input is complete. • Spontaneous speech including disfluencies and repair phenomena. • Speaker adaptive. • Mixed initiative dialogue • Three different domains of discourse CLINT-CS Verbmobil

  5. Domains CLINT-CS Verbmobil

  6. Data Collection A signficant programme of data collection was performed To extract statistical properties of different kinds of data Transliterated speech data Treebanks & predicate argument structures Segmented speech with prosodic labels Aligned bilingual Corpora Dialogues annotated with dialogue acts CLINT-CS Verbmobil

  7. Speech Data • Multi channel recording • close-speaking microphone • room microphone • various telephones • Speech recognisers trained on data sets of different audio quality CLINT-CS Verbmobil

  8. Multi Level Data Annotation • Speech Data • Transliteration • Orthography • Pronunciation • Phonological Segmentation • Word Segmentation • Prosodic Segmentation • Non Speech • Dialogue Acts • Treebanks CLINT-CS Verbmobil

  9. Statistical Models • Data used to train different statistical models using Machine Learning. • Models include • Neural Networks • Probabilistic Automata (HMMs for speech) • Probabilistic CFGs (robust parsing) • Probabilistic Transfer Rules CLINT-CS Verbmobil

  10. CLINT-CS Verbmobil

  11. Architecture • Different input devices (microphone, telephone, mobile, internet) • Multilingual speech recognition (EN, DE, JP) including prosodic analysis • Parsing • Multi-level translation • Multi-lingual generation CLINT-CS Verbmobil

  12. Multi Engine Parsing Architecture • Three different parsing models are employed • Probabilistic LR Parser • Robust Chunk Parsing • HPSG Chart Parser • All parsing models produce trees that are tranformed into the same multistratal representation called VIT (Verbmobil Interface Terms) • This facilitates integration of partial results from the different parsing models CLINT-CS Verbmobil

  13. Translation Models • Substring Based • Template Based • Dialogue Act Based CLINT-CS Verbmobil

  14. Substring Based Translation • Starts with the best sentence hypothesis of the speech recogniser • Uses prosodic information to determine phrase boundaries and sentence mode • Machine Learning methods applied to a sentence-aligned bilingual corpus • The output of this module is a sequence of words in the target language together with a confidence measure that is used for selecting the best translation. CLINT-CS Verbmobil

  15. Template Based Translation • Based on 30K translation templates learned from a sentence-aligned corpus Ti = (Tis,Tit){x1,..,xn} • 3 phases: • SL Template matching • Subphrase Translation • TL utterance generation CLINT-CS Verbmobil

  16. Template Translation Results CLINT-CS Verbmobil

  17. Multi Engine Translation Segment 1 If you prefer another hotel Segment 2 please let me know case based translation substring based translation statistical translation dialogue based translation semantic transfer selection module Segment 1 Semantic Xfer Segment 2 CBT CLINT-CS Verbmobil

  18. Dialogue Act Based Translation • Meaning based translation • Statistical classification of 19 dialogue acts. • Extraction of propositional content using finite state transducers. • Content built from an ontology covering appointment scheduling and travel planning tasks. • Template based approach to generation of target language from content. CLINT-CS Verbmobil

  19. Part of Ontology for Propositional Content top object situation quality event action abstract concrete journey move stay show meeting agent location move by public transport move-by-rail move-by-plane CLINT-CS Verbmobil

  20. Dialogue Act Hierarchy greet bye control dialogue introduce thank deliberate init defer close request suggest request clarify request comment request commit Dialogue Act promote task request suggest inform feedback commit offer manage task digress exclude clarify justify CLINT-CS Verbmobil

  21. Dialogue –Based Translation:Transfer Component rules Semantic Representation Source Language VIT Dialogue and context evaluation Semantic Representation Target Language VIT GENERATION CLINT-CS Verbmobil

  22. Prosody • Input • Speech signal • Word Hypothesis Graph (WHG) • Output • annotated WHG including, per word • duration, pitch, energy, pause info • Used to classify phrase and clause boudaries, accented words, and sentence mood. CLINT-CS Verbmobil

  23. Prosody – Sentence Mood pitch row? mor You are coming to time You are coming to mor ro w. CLINT-CS Verbmobil

  24. Use of Prosodic Information • Prosodic information is used systematically at all processing stages • Prosodic difference can lead to different translation… wir haben noch (we still have vs. we have another) CLINT-CS Verbmobil

  25. Multi Blackboard Architecture • Final system comprises 69 highly interactive modules. • No direct communication between modules. • Communication is handled by 198 blackboards. • Shared representation structures • A module typically subscribes to several blackboards. CLINT-CS Verbmobil

  26. Blackboards & Modules command recogniser speaker adaptation Audio Data spontaneous speech recogniser prosodic analysis statisstical parser chunk parser WHG with prosodic labels dialogue act recognition HPSG parser semantic construction semantic transfer VIT discourse representation robust dialogue semantics generation CLINT-CS Verbmobil

  27. Multi Engine Approach Augmented WHG statistical parser chunk parser HPSG parser chart containing partial VITs robust dialogue semantic KBased reconstruction complete and spanning VIT CLINT-CS Verbmobil

  28. Achievements • 3 language pairs, three domains and a vocalbulary size of over 100K word forms • Average processing time 4x original signal duration • Word recognition rate of 75% for spontaneous speech • 80% approximately correct translations • 90% success rate for dialogue tasks in end-to-end evaluation CLINT-CS Verbmobil

  29. Conclusion • Speech to speech translation of spontaneous dialogues can only be cracked by combining deep and shallow processing • The final architecture maximises the necessary interaction between processing modules • Software engineering considerations must be taken seriously in such a project. CLINT-CS Verbmobil

More Related