1 / 47

Introduction to Voice Conversion

Introduction to Voice Conversion. Hsin-Te Hwang max0219.cm94g@nctu.edu.tw Department of Communication Engineering, Chiao Tung University, Hsinchu. Outline. Introduction VC baseline (GMM based VC) Problems Summary References. What is voice conversion (VC)?. Definition:

lars
Download Presentation

Introduction to Voice Conversion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Introduction to Voice Conversion Hsin-Te Hwang max0219.cm94g@nctu.edu.tw Department of Communication Engineering, Chiao Tung University, Hsinchu

  2. Outline • Introduction • VC baseline (GMM based VC) • Problems • Summary • References

  3. What is voice conversion (VC)? • Definition: • To modify the speech signal of one speaker (source) to sound like the other speaker (target). • More generalized definition: • To modify (transform) the characteristics of the speech signal. Ex: Emotional Voice Conversion [1,2]

  4. Application of VC • In TTS: • Building a new voice based on Current state of the art TTS system such as Corpus based TTS is hard. • Same problem in building an Emotional TTS [1,2]. • By using VC, one can use recorded database and convert it to a target voice using as little as 10-20 sentences [3]. • Others: • To convert narrow-band speech to wide band speech for telecommunication [4]. • Modeling of speech production [5].

  5. Conversion? • Spectrum: • Convert Spectrum only. Prosody remains unchanged or uses sample way to convert prosody. • Prosody • Convert prosody only. • Spectrum + Prosody • Convert spectrum and prosody.

  6. Overview of Techniques • Abe et al. (1988) [6]: VQ mapping • Valbret et al. (1992) [7]: Linear Multivariate Regression (LMR). Dynamic Frequency Warping (DFW) • Kuwabara et al. (1995) [8]: Fuzzy VQ • M. Narendranath et al. (1995) [9]: ANN based • Stylianou et al. (1995) [10]: GMM based • Kain et al. (1998) [11]: GMM based • Toda et al. (2001) [12]: GMM and DFW • Toda et al. (2005) [13]: GMM consider Globe Variance • Mouchtaris et al. (2006) [14]: GMM and speaker adaptation

  7. Outline • Introduction • VC baseline (GMM based VC) • Problems • Summaries • Reference

  8. The block diagram for building VC system. • The following figure shows the block diagram of a voice conversion system.

  9. Review GMM based VC • Start form Minimum Mean Square Estimation (MMSE) • Time alignment • To derive the transfer function of GMM based VC.

  10. Mean-Square Estimation(1/4)

  11. Mean-Square Estimation(2/4)

  12. Mean-Square Estimation(3/4)

  13. Mean-Square Estimation(4/4)

  14. Stylianou-GMM based mapping function (1/2) Probability classification: • Modeling acoustic space of source speaker by using GMM • Classification:

  15. Stylianou-GMM based mapping function (2/2) • Mapping Function [10]: • Motivation: • Estimation of mapping function:

  16. Parallel data time alignment using DTW (1/2)

  17. Parallel data time alignment using DTW (2/2)

  18. Kain-GMM based mapping function

  19. Stylianou based vs Kain based VC • Kain[11] based method makes no assumptions about the target distributions: clustering takes place on the source and the target vectors. • In theory, modeling the joint density rather than the source density should lead to a more judicious allocation of mixtures for the regression problem. • Kain based method is computationally more expensie during the EM step than Stylianou [10].

  20. Outline • Introduction • VC baseline (GMM based VC) • Problems • Summary • Reference

  21. Problems • To make the training more flexible (non-parallel training) • To improve the quality and similarity of transform speech • Prosody conversion • Other issues

  22. Problems of parallel training for VC • In order to derive the conversion function, a speech corpus is needed that contains the same utterances form both the source and target speakers. Such corpus is called parallel corpus. • The disadvantage of this method is that such corpus is difficultor even impossible to collect. • Cross lingual voice conversion. • Most of the databases are nonparallel.

  23. Nonparallel training for VC • Mouchtaris et al. (2004, 2006) [14,15]: GMM and speaker adaptation • D. Säundermann et al (2003) [16] VTLN based • H. Ye et al (2004) [17] VC for Unknown Speaker • M. Mashimo et al. (2001) [18] Cross-Language VC

  24. Nonparallel Training for Voice Conversion by ML Constrained Adaptation (1/2) Mouchtaris et al. (2004, 2006) [14,15]: Assuming: • Parallel data for two speakers exist • Conversion function between these two speakers is known Then: • Adapt S1 to the Source speaker • Adapt S2 to the Target speaker • Compute Conversion function by using: • The initial conversion function of the parallel data • The adaptation parameters

  25. Nonparallel Training for Voice Conversion by ML Constrained Adaptation (2/2) Block diagram of nonparallel VC [14,15]

  26. Quality improvement Two major problems of GMM based VC: • Time independent assumption • Over-smooth

  27. Time independent assumption(1/2) • GMM based mapping function performs the frame by frame basis. ( Time independent approach). • The correlation of the target feature vectors between frames is ignored in the conventional mapping.

  28. Time independent assumption(2/2) Example of converted and natural target parameter trajectories. [24]

  29. Solution for time independent assumption (1/3) Duxans et al [23] (HMM based voice conversion): • HMM are well-known models which can capture the dynamics of the training data using states. • it can model the dynamics of sequences of vectors with transition probabilities between states. HMM based VC system block diagram [23]

  30. Solution for time independent assumption (2/3) Chi-Chun Hsia et al [21] (Gaussian Mixture Bi-gram Model): • To Adopt the Gaussian mixture bi-gram model to characterize temporal and spectral evolution in the conversion function.

  31. Solution for time independent assumption (3/3)

  32. Over-smooth problem (1/3)

  33. Over-smooth problem (2/3)

  34. Over-smooth problem (3/3) Example of converted and natural target spectra. [24]

  35. Solutions for over-smooth problem (1/2)

  36. Solutions for over-smooth problem (2/2) Toda et al [11,29]: • Combine joint GMM with the global variance of the converted spectra in each utterance to cope with over-smoothing • Use of delta features have been used to alleviate spectral discontinuities

  37. CART based voice conversion(1/2) Duxans et al [23]: • UsingGMMor HMM, we only have spectral informationto identify the classes. But using decision trees we canalso use phonetic information. • Phonetic information for each frame, such as the phone, a vowel/consonant flag, point of articulation, manner and voicing.

  38. CART based voice conversion(2/2) • Multiple conversion functions • Improve the performance of conversion • GMMbased vs HMM based vs CART based

  39. Prosody conversion • Chi-Chun Hsia, Chung-Hsien Wu,(2007) [21] “A Study on Synthesis Unit Selection and Voice Conversion for Text-to-Speech Synthesis” • Hanzlíček, Zdeněk et al (2007) [22] "F0 transformation within the voice conversion framework” • Guoyu Zuo et al (2005) [19] “ Mandarin Voice Conversion Using Tone Codebook Mapping. • E.E.Helander et al (2007) [2] “A Novel Method for Prosody Prediction in Voice Conversion”

  40. Other issues • Subjective and objective evaluation • Cross-lingual voice conversion [25] • Time alignment • A novel VC frame work [26] • Residual prediction [27]

  41. Summary • To increase the usefulness of the voice conversion system, practical aspects should be considered. • Flexible training framework • Quality and Similarity • Objective Evaluation

  42. References (1/5) [1] Chung-Hsien Wu, Chi-Chun Hsia, Te-Hsien Liu, and Jhing-Fa Wang, “Voice Conversion Using Duration-Embedded Bi-HMMs for Expressive Speech Synthesis, IEEE Trans. Audio, Speech and Language Processing, vol. 14, no. 4, July, 2006, pp. 1109-1116. [2] Chi-Chun Hsia, Chung-Hsien Wu, Jian-Qi Wu, “Conversion Function Clustering and Selection Using Linguistic and Spectral Information for Emotional Voice Conversion, “ IEEE Trans. Computers (Special Issue on Emergent Systems, Algorithms and Architectures for Speech-based Human machine Interaction), vol. 56, no. 9, September 2007, pp. 1225-1233. [3] http://festvox.org/transform/transform.html [4] K. Y. Park and H. S. Kim, “Narrowband to wideband conversion of speech using GMM based transformation,” in Proc. ICASSP, Istanbul, Turkey, Jun. 2000, pp. 1847–1850. [5] K. Richmond, S. King, and P. Taylor, “Modelling the uncertainty in recovering articulation from acoustics,” Comput. Speech Lang., vol. 17, pp. 153–172, 2003. [6] M. Abe, S. Nakamura, K. Shikano and H. Kuwabara, “Voice conversion through vector Quantization,”in Proc. of ICASP, New York, NY, USA, pp. 655-658, Apr. 1988. [7 ] N. Iwahashi and Y. Sagisaka, “ Speech spectrum transformation based on speaker interpolation.” in Proc. ICASSP94. 1994.

  43. References (2/5) [8] H. Kuwabara and Y. Sagisaka, “ Acoustic characteristics of speaker individuality: Control and conversion, “ Speech Communication, vol,19, no. 2, pp. 165-173, 1995. [9] M. Narendranath, H. A. Murthy, S. Rajendran, and B. Yegnanarayana, “Transformation of formants for voice conversion using artificial neural networks,” Speech Commun., vol. 16, no. 2, pp. 207–216, 1995. [10] Y. Stylianou, “Continuous probabilistic transform for voice conversion,”IEEE Trans. on Speech and Audio Processing, vol. 6, no. 2, pp. 131-142, Mar. 1998. [11] A. Kain and M. W. Macon, “Spectral Voice Conversion for Text-to-Speech Synthesis,” in Proc. of ICASSP, vol. 1, pp. 285-288, Seattle, Washington, USA, May 1998. [12] T. Toda, H. Saruwatari, and K. Shikano, “Voice Conversion Algorithm based on Gaussian Mixture Model with Dynamic Frequency Warping of STRAIGHT spectrum, “in Proc. IEEE Int. Conf. Acoust, Speech, Signal Processing, (Salt Lake City, USA), pp. 841-844,2001. [13] T. Toda, A. Black, and K. Tokuda, “ Spectral Conversion Based on Maximum Likelihood Estimation considering Global Variance of Converted Parameter,” in Proc. IEEE Int. Conf. Acoust. Speech, Signal Processing, (Philadelphia, USA), pp. 9-12, 2005. [14] A. Mouchtaris, J. Van der Spiegel, and P. Mueller, “Non-Parallel Training for Voice Conversion Based on a Parameter Adaptation Approach”, in IEEE Trans. Audio, Speech and Language Processing, vol. 14, no. 3, May 2006, pp. 952-963.

  44. References (3/5) [15] A. Mouchtaris, J. Spiegel, and P. Mueller, Non-Parallel Training for Voice Conversion by Maximum Likelihood Constrained Adaptation," in Proc: of the ICASSP'04, Montreal, Canada, 2004. [16] D. SÄundermann, H. Ney, and H. HÄoge, VTLN-Based Cross-Language Voice Conversion," in Proc: of the ASRU'03, St:Thomas, USA, 2003. [17] H. Ye and S. J. Young, \Voice Conversion for Unknown Speakers," in Proc: of the ICSLP'04, Jeju, South Korea, 2004. [18] M. Mashimo, T. Toda, K. Shikano, and N. Campbell, Eval-uation of Cross-Language Voice Conversion Based on GMMand STRAIGHT," in Proc: of the Eurospeech'01, Aalborg,Denmark, 2001. [19] Guoyu Zuo, Yao Chen, Xiaogang Ruan, Wenju Liu: MandarinVoiceConversion Using ToneCodebookMapping. ICMLC 2005: 965-973 [DBLP:conf/icmlc/ZuoCRL05] [20] E.E.Helander,J.Nurminen.2007.A Novel Method for Prosody Prediction in Voice Conversion Acoustics.Speech and Signal Processing.ICASSP 2007.IEEE International Conference on Volume 4:509-512 [22] Hanzlíček, Zdeněk / Matoušek, Jindřich (2007): "F0 transformation within the voice conversion framework", In INTERSPEECH-2007, 1961-1964.

  45. References (4/5) [21] Chi-Chun Hsia, Chung-Hsien Wu, “A Study on Synthesis Unit Selection and Voice Conversion for Text-to-Speech Synthesis”, Department of Computer Science and Information Engineering, NCKU, Dissertation for Doctor of Philosophy, December 2007. [23]Duxans, H., Bonafonte, A., Kain, A. and van Santen, J., “Including Dynamic and Phonetic Information in Voice Conversion Systems,” in Proc. of ICSLP 2004, pp. 5-8, Jeju Island, South Korea, 2004. [24]T. Toda, A.W. Black, K. Tokuda, ''Voice Conversion Based on Maximum Likelihood Estimation of Spectral Parameter Trajectory,'' IEEE Transactions on Audio, Speech and Language Processing, Vol. 15, No. 8, pp. 2222-2235, Nov. 2007. [25] D. S¨undermann, H. Ney, and H. H¨oge, “VTLN-Based Cross-Language Voice Conversion,” in Proc. of the ASRU’03, Virgin Islands, USA, 2003. [26] T. Toda, Y. Ohtani, and K. Shikano, “One-to-many and many-to-one voice conversion based on eigenvoices,” in Proc. ICASSP, Honolulu, HI, Apr. 2007, vol. 4, pp. 1249–1252. [27] A. Kain and M. W. Macon, “Design and evaluation of a voice conversion algorithm based on spectral envelope mapping and residual prediction,” in Proc. ICASSP, Salt Lake City, UT, May 2001, pp. 813–816.

  46. References (5/5) [28] L. Meshabi, V. Barreaud, and O. Boeffard, “GMM-based Speech Transformation Systems under Data Reduction,” 6th ISCA Workshop on Speech Synthesis, pp.119-124. August 22-24, 2007. [29]T. Toda, A.W. Black, K. Tokuda, “ Voice Conversion Based on Maximum Likelihood Estimation of Spectral Parameter Trajectory,'' IEEE Transactions on Audio, Speech and Language Processing, Vol. 15, No. 8, pp. 2222-2235, Nov. 2007.

  47. Thanks for your listening! Q&A?

More Related