Violin Pedagogy for Finger and Bow Placement using Augmented Reality

1. Violin Pedagogy for Finger and Bow Placement using Augmented Reality Francois de Sorbier Hiroyuki Shiino Hideo Saito

2. Outline • Introduction • Overview of our system • Violin extraction and 3D registration • Virtual advising • Conclusion Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

3. Introduction Motivation • Violin is a beautiful instrument… • … butone of the most complicated • No fret on the fingerboard • No help for the position the bow on strings Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

4. Introduction Previous works • Music Jacket • Vibro-tactile feedback • Guide the bowing arm • Guitar playing support • Tracking with marker • Guide with virtual hand van der Linden, J., Schoonderwaldt, E. and Bird, J. “Good Vibrations: Guiding Body Movements with VibrotactileFeedback”.Proceedings of the Third International Workshop on Physicality, 13-18, 2009 Y. Motokawa, H. Saito. “Support system for guitar playing using augmented reality display”. In Proceedings of the 5th IEEE and ACM ISMAR, 243-244, 2006 Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

5. Introduction Our objective • Overlay virtual information on the violin • Virtual frets • Guides for the bow and fingers • Sound analysis Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

6. Introduction Our constraints • No intrusive device • No marker • Real time feedback Marker-free violin tracking using a RGBD camera Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

7. Overview of our system Overview of our system • Tracking of the violin • Feedback displayed on the screen • Constant view of the violin • Virtual information Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

8. Overview of our system Marker less tracking • Features detection and extraction • Many occlusions caused by the player • The surface has a poor texture • The material is highly specular • Difficult to use features in this context Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

9. Overview of our system Depth-based pose estimation • Use Kinect for tracking the violin • Depth values for the pose estimation Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

10. Overview of our system Workflow Color Segmentation Database Depth Violin detection Violin extraction Registration Virtual information displayed Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

11. Violin extraction and 3D registration Color segmentation • Detect the brown color in the image • Remove noise • Many parts are missing • Occlusions • Specular material • Strings and fingerboard • Not enough for tracking Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

12. Violin extraction and 3D registration Extended segmentation • Get the 3D points from extracted color • Compute the corresponding plane equation • Optimized with RANSAC • Align a 3D volume to the plane • Typical dimensions of a violin • Keep the 3D points in it Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

13. Violin extraction and 3D registration 3D registration with ICP • Iterative Closest Point algorithm • Compare input points with a model • Slow if too many points • Inaccurate if not enough • Our proposed solution • Increase the number of models • Reduce the number of points per model • Fast retrieval with a plane equation comparison Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

14. Violin extraction and 3D registration Creating the database • Offline phase • 25 models • Compare the plane equations • Store candidate if the difference is big enough • Store also the plane equation Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

15. Violin extraction and 3D registration Result of the tracking Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

16. Violin extraction and 3D registration Evaluation of the tracking • Real time (21 milliseconds) • Pose also estimated using markers Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

17. Virtual advising Display the frets • Use the pose estimation • Location manually defined during the capture of the models Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

18. Virtual advising Fingering advices • Emphasize the string to be pressed • Display a red dot at the junction of the string and the fret • Define where the finger has to press the string Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

19. Virtual advising Other advices • Suggest the violinist to play a given note • Display the fret/string to be pressed • Analyze the sound obtained • If fingering is considered correct • Advice about the position of the bow given the difference of pitch (OK / LOW / HIGH) Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

20. Virtual advising Results Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

21. Virtual advising Evaluation of the results • Performed by confirmed player • Bowing is correct • Compute the difference of pitch on each fret with the expected one Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

22. Virtual advising Conclusions • Original marker-free method for virtual advising on a violin • Method based on several pre-computed models • Real-time • Accurate • Display virtual guides on the fingerboard • Analyze the note played for further advices Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

23. Future works • Perform a user based analysis • Validate or improve our approach • Use different kind of display • See-through HMD • Spatial augmented reality • Apply to other similar instruments • Japanese shamisen Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier

24. Questions Thank you for your attention Violin Pedagogy for Finger and Bow Placement using Augmented Reality - F. de Sorbier