Raul Fernandez and Bhuvana Ramabhadran I.B.M. T.J. Watson Research Center

1. Automatic Exploration of Corpus-SpecificProperties for Expressive Text-to-Speech. (A Case Study in Emphasis.) Raul Fernandez and Bhuvana Ramabhadran I.B.M. T.J. Watson Research Center Sixth Speech Synthesis Workshop, Bonn, Germany.

2. Outline • Motivation. • Review of Expressive TTS Architecture • Expression Mining: Emphasis. • Evaluation. Sixth Speech Synthesis Workshop, Bonn, Germany.

3. Expressive TTS We have shown that corpus-based approaches to expressive CTTS manage to convey expressiveness if the corpus is well designed to contain the desired expression(s). There are, however, shortcomings to this approach: • Adding new expressions, or increasing the size of the repository for an existing one, is expensive and time consuming. • The footprint of the system increases as we add new expressions. Without abandoning this framework, we propose to partially address these limitations by an approach that exploits the properties of the existing databases to maximize the expressive range of the TTS system. Sixth Speech Synthesis Workshop, Bonn, Germany.

4. Some observations about data and listeners… • Production variability: • Speakers produce subtle expressive variations, even when they’re asked to speak in a mostly-neutral style. Sixth Speech Synthesis Workshop, Bonn, Germany.

5. Neutral Anger Fear Sad Some observations about data and listeners… • Production variability: • Speakers produce subtle expressive variations, even when they’re asked to speak in a mostly-neutral style. • Perceptual confusability/redundancy: • Several studies have shown that there’s an overlap in the way listeners interpret the prosodic-acoustic realizations of different expressions. Sixth Speech Synthesis Workshop, Bonn, Germany.

6. Neutral Anger Fear Sad Some observations about data and listeners… • Production variability: • Speakers produce subtle expressive variations, even when they’re asked to speak in a mostly-neutral style. • Perceptual confusability/redundancy: • Several studies have shown that there’s an overlap in the way listeners interpret the prosodic-acoustic realizations of different expressions. Sixth Speech Synthesis Workshop, Bonn, Germany.

7. Expression Mining • Goals: • Exploit the variability present in a given dataset to increase the expressive range of the TTS engine. • Augment the corpus-based with an expression-mining approach for expressive synthesis. • Challenge: • Automatic annotation of instances in the corpus where an expression of interest occurs. • (Approach may still require collecting a smaller expression-specific corpus to bootstrap data-driven learning algorithms.) • Case study: Emphasis. Sixth Speech Synthesis Workshop, Bonn, Germany.

8. Outline • Motivation. • Review of Expressive TTS Architecture • Expression Mining: Emphasis. • Evaluation. Sixth Speech Synthesis Workshop, Bonn, Germany.

9. The Expressive Framework of the IBM TTS System • The IBM Expressive Text-to-Speech consists of: • a rules-based front-end for text analysis • acoustic models (DTs) for generating synthesis candidate units • prosody models (DTs) for generating pitch and duration targets • a module to carry out a Viterbi search • a waveform generation module to concatenate the selected units • Expressiveness is achieved in this framework by associating symbolic attribute vectors with the synthesis units. These attribute values are able to influence the • target prosody generation • unit-search selection Sixth Speech Synthesis Workshop, Bonn, Germany.

10. Attributes Style … Uncertain Apologetic Good News Default Attribute Sixth Speech Synthesis Workshop, Bonn, Germany.

11. Attributes Emphasis 1 0 Style … Uncertain Apologetic Good News Sixth Speech Synthesis Workshop, Bonn, Germany.

12. Attributes Emphasis 1 0 Style … Uncertain Apologetic Good News ? (e.g., voice quality={breathy,…}, etc.) Sixth Speech Synthesis Workshop, Bonn, Germany.

13. How do attributes influence the search? • Corpus is tagged a priori. • At run time: Input is tagged at the word level (e.g., via user-provided mark-up) with annotations indicating the desired attribute. Annotations are propagated down to the unit level. • A component of the target cost function penalizes label substitutions: Sixth Speech Synthesis Workshop, Bonn, Germany.

14. How do attributes influence the search? • Additionally, the style attribute has style-specific prosody models (for pitch and duration) associated with it. Therefore, prosody targets are produced according to the style requested. Prosody Model Style 1 Prosody Model Style 2 Prosody Targets Model Output Generation Normalized Text Prosody Model Style 3 Target Style Sixth Speech Synthesis Workshop, Bonn, Germany.

15. Outline • Motivation. • Review of Expressive TTS Architecture • Expression Mining: Emphasis. • Evaluation. Sixth Speech Synthesis Workshop, Bonn, Germany.

16. Mining Emphasis Baseline Corpus (~10K sents.) Emphasis Corpus (~1K sents.) Baseline Corpus w. Emphasis Labels Trained Emphasis Classifier Statistical Learner Build TTS System w. Emphasis Sixth Speech Synthesis Workshop, Bonn, Germany.

17. Training Materials • Two sets of recordings, one from a female and one from a male speaker of US English. • Approximately 1K sentences in script. • Approximately 20% of words in script contain emphasis. • Recordings are single channel, 22.05kHz. Exs: • To hear DIRECTIONS to this destination say YES. • I'd LOVE to hear how it SOUNDS. • It is BASED on the information that the company gathers, but not DEPENDENT on it. Sixth Speech Synthesis Workshop, Bonn, Germany.

18. Interm. Output Probs. Prosodic Features K-Nearest Neighbor Naïve Bayes Final Output Probs. SVM Modeling Emphasis – Classification Scheme • Modeled at the word level. • Feature set: prosodic features derived from (i) pitch (absolute; speaker-normalized), (ii) duration, and (iii) energy measures. • Individual classifiers are trained, and results stacked (this marginally improves the generalization performance estimated through 10-fold CV). Sixth Speech Synthesis Workshop, Bonn, Germany.

19. Modeling Emphasis – Classification Results TP Rate FP Rate Prec. F-Meas. Class 0.82 0.06 0.78 0.80 emphasis 0.94 0.18 0.95 0.94 notemphasis Correctly Classified Instances 91.2 % M A L E F E M A L E TP Rate FP Rate Prec. F-Meas. Class 0.80 0.06 0.75 0.77 emphasis 0.93 0.18 0.94 0.94 notemphasis Correctly Classified Instances 89.9 % Sixth Speech Synthesis Workshop, Bonn, Germany.

20. What does it find in the corpus? Sixth Speech Synthesis Workshop, Bonn, Germany.

21. What does it find in the corpus? I think they will diverge from bonds, and they may even go up. Sixth Speech Synthesis Workshop, Bonn, Germany.

22. What does it find in the corpus? Sixth Speech Synthesis Workshop, Bonn, Germany.

23. What does it find in the corpus? Please say the full name of the person you want to call. Sixth Speech Synthesis Workshop, Bonn, Germany.

24. What does it find in the corpus? Sixth Speech Synthesis Workshop, Bonn, Germany.

25. What does it find in the corpus? There's a long fly ball to deep center field. Going, going. It's gone, a home run. Sixth Speech Synthesis Workshop, Bonn, Germany.

26. Outline • Motivation. • Review of Expressive TTS Architecture • Expression Mining: Emphasis. • Evaluation. Sixth Speech Synthesis Workshop, Bonn, Germany.

27. Listening Tests – Stimuli and Conditions Synthesis Sources Target Condition 1 Pair: 1 Type-A sentence vs. 1 Type-B sentence (in random order). Condition 2 Pair: 1 Type-A sentence vs. 1 Type-C sentence (in random order). Sixth Speech Synthesis Workshop, Bonn, Germany.

28. B1 vs A1 A2 vs B2 A3 vs B3 … B12 vs A12 + A1 vs C1 A2 vs C2 C3 vs A3 … C12 vs A12 Listening Tests – Setup A2 vs C2 B1 vs A1 B12 vs A12 … A3 vs B3 L I S T 1 Condition 1 (12 Pairs) Shuffle Reverse Order Pair C2 vs A2 A1 vs B1 A12 vs B12 … B3 vs A3 L I S T 2 Condition 2 (12 Pairs) Sixth Speech Synthesis Workshop, Bonn, Germany.

29. Listening Tests – Task Description • A total of 31 participants listen to a playlist (16 to List 1; 15 to List 2) • For each pair of stimuli, the listeners are asked to select which member of the pair contains emphasis-bearing words • No information is given about which words may be emphasized. • Listeners may opt to listen to a pair repeatedly. Sixth Speech Synthesis Workshop, Bonn, Germany.

30. Listening Tests – Results Sixth Speech Synthesis Workshop, Bonn, Germany.

31. Conclusions • When only the limited expressive corpus is considered, listeners actually prefer the neutral baseline. Possible explanation is that biasing the search heavily toward a small corpus is introducing artifacts that interfere with the perception of emphasis. • However, when the small expressive corpus is augmented with automatic annotations, the perception of intended emphasis increases significantly by 13% (p<0.001). • Although further work is needed to reliably convey emphasis, we have demonstrated the advantages of automatic mining the dataset to augment the search space of expressive synthesis units. Sixth Speech Synthesis Workshop, Bonn, Germany.

32. A A N GN GN U U N Future Work • Explore alternative feature sets to improve automatic emphasis classification. • Extend the proposed framework to automatically detect more complex expressions in a “neutral” database and augment the search space for our expressive systems (e.g., good news; apologies; uncertainty) • Explore how the perceptual confusion between different labels can be exploited to increase the range of expressiveness of the TTS system. Sixth Speech Synthesis Workshop, Bonn, Germany.

33. Automatic Exploration of Corpus-SpecificProperties for Expressive Text-to-Speech. (A Case Study in Emphasis.) Raul Fernandez and Bhuvana Ramabhadran I.B.M. T.J. Watson Research Center Sixth Speech Synthesis Workshop, Bonn, Germany.