1 / 9

Error Analysis: Indicators of the Success of Adaptation

Error Analysis: Indicators of the Success of Adaptation. Arindam Mandal, Mari Ostendorf, & Ivan Bulyko University of Washington. Goals and Motivation. Compare WER1 and WER2 Identify situations where MLLR breaks (then later try to improve these cases)

fritz-hood
Download Presentation

Error Analysis: Indicators of the Success of Adaptation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Error Analysis:Indicators of the Success of Adaptation Arindam Mandal, Mari Ostendorf, & Ivan Bulyko University of Washington

  2. Goals and Motivation • Compare WER1 and WER2 • Identify situations where MLLR breaks (then later try to improve these cases) • Understand the role of WER vs. speaker characteristics speech 1st pass recognizer MLLR 2nd pass recognizer new hyp hyp + conf WER2 WER1

  3. Some Possible Analyses • Predict relative WER reduction at the speaker level: (WER1-WER2)/WER1 • Predict relative WER reduction at the utterance level • Predict error correction (and introduction) at the word level Our current focus is the speaker level.

  4. Experimental Paradigm • Data • Train: Eval98-02 + Eval03 (Fisher set) [472 speakers] • Test: Eval03 (swbd set) [72 speakers] • System: SRI Aug 03, mfcc sub-system • 1st pass: phone-loop adapt + rescore* • 2nd pass: unsupervised MLLR + rescore* * Rescoring with 4-gram LM, duration model, pron model & pause model

  5. WER Gains for Different Speakers More than 15% of the speakers take a hit when we use MLLR!

  6. Prediction Approach • Model: • Multiple linear regression • Mixed forward & backward feature selection using cross-validation • Features • Various functions of confidence before/after rescore • Mean and variance of rate of speech for this speaker • Speaker VTL warp factor • F0 mean/sigma, F0 variation • Energy mean/sigma, energy variation

  7. Results • Mean % change for test data = 6.9 • RMSE predicting % change in WER

  8. Results (cont.) • Since overall prediction is not great, we looked at the “goats” (negative % change) • Regression prediction does much worse for these speakers! (9.2 vs. 7.9 rmse) • Can we predict “sheep vs. goats”? • Somewhat better than WER change… 33% error relative to 50% chance

  9. Summary • Some observations • Confidence is the single most important factor; WER alone does not explain changes • Easier to predict categories of speakers than relative WER change • Future work: • New acoustic-prosodic features • Analyses aimed at how to use confidence in adaptation

More Related