Multimodal Interface for Mobile Voice Search Recovery: A Review

1. LEAD REVIEW ON THE PAPERSearch Vox: Leveraging Multimodal Refinement and Partial Knowledge for Mobile Voice SearchbyTim Peak et al. Ivan Elhart - ECE 992 Ubiquitous Computing University of New Hampshire 10/09/2008

2. Problems • Mobile settings often contain non-stationary noise which cannot be easily canceled • Speakers tend to adopt to surrounding noise in acoustically unhelpful ways Goal • New multimodal interface that will help voice search applications to recover from speech recognition errors Hypothesis • A multimodal interface for mobile voice search that incorporates speech with touch and text may increase recovery rates of the search

3. Background • Yu, D., Ju, Y.C., Wang, Y.Y., Zweig, G., & Acero, A. 2007. Automated directory assistance system: From theory to practice. Proc. of Interspeech. • Ainsworth, W.A. & Pratt, S.R. 1992. Feedback strategies for error correction in speech recognition systems. International Journal of Man-Machine Studies, 26(6), 833-842. • Hsu, P., Mahajan, M. & Acero, A. 2005. Multimodal text entry on mobile devices. Proc. of ASRU.

4. Approach • N –best list, whenever recognition is less than perfect • Coupling of speech with touch and text • Leveraging of any partial knowledge

5. Approach • Word palette • Allows users to select any word of a recognized phrase

6. Approach • Text hints • Resort to speech whenever search entry is too long or when enough text hints have been provided

7. Approach • Verbal wildcards • Partial knowledge queries

8. Approach • Search Vox architecture

9. Results • Simulation experiments • Utterances collected from Microsoft Live Search Mobile • Automated directory assistance, maps, driving directions, movie times, local gas prices

10. Results • Word palette

11. Results • Text hints

12. Conclusion

13. LEAD REVIEW ON THE PAPER“It’s Mine, Don’t Touch!”: Interactions at a large Multi-Touch Display in a City CentrebyPeter Peltonen et al. Ivan Elhart - ECE 992 Ubiquitous Computing University of New Hampshire 10/07/2008

14. Problem • How does the outdoor public tangible interface support simultaneous participation and interaction of multiple users? Goal • To provide first insights into how users approach, participate, and interact on a large multi-touch display in a public space Hypotheses • Observational studies in urban environments could help in understanding how multi-touch screens can affect and support social interactions • Public interactive multi-touch displays can potentially restructure the way people experience and use the space around them

15. Background • Semi-public displays

16. Background • Public displays

17. Approach City Wall Direct manipulation Installation in Helsinki Non-modality

18. Approach • Data collection • A continuous interaction log was written • Interactions were recorded with a web camera in a 640x480 resolution • Twelve on-site interviews were conducted • Data analysis • Combination of the video and interaction log • First time and returning users • Sessions (a ten second gap between interaction) • Duration • Number of active users • Number of passive bystanders

19. Results • Findings on how the City Wall was used and how the users interacted with each other at the screen • 8 days of interaction, 8.8% the display was used, 1199 users, 516 sessions, and 202 passive bystanders 72% 23% 18%

20. Results • Noticing the Display (when the wall was used) • Multi-user interactions (fun to use with others and friends) Rain shelter Stepwise app. Parallel use Team work

21. Results • Conflict management • Social configurations (or roles) Social inter. Withdrawal Comedian Teacher Leaving a mark

22. Conclusion Noticing City Wall Installation Parallel use Conflicts