Topic 10: Motion Tracking Part II

1. Topic 10: Motion Tracking Part II SCM-CityU

2. Clipping Regions • By default, motion tracking is done over the entire video domain • Check out MotionTracker-Skeleton.fla SCM-CityU

3. Clipping Regions SCM-CityU

4. Clipping Regions • What if motion only within specific regions (called clipping regions) are needed? • Motion outside those clipping regions will be ignored • Demo clipping regions SCM-CityU

5. Clipping Regions • Specify a clipping region when creating a motion tracker • cl, ct, cr, cb are with respect to top-left cornerof webcam • trackX, trackYare also withrespect to top-left cornerof webcam • You can have only ONE clipping region for one tracker tracker =newMotionTracker(webcam, cl,ct,cr,cb); ct cb cl cr Clipping Region SCM-CityU

6. Multiple Clipping Regions • To have multiple clipping regions, you can create multiple trackers • All trackers share the same Webcam instance • Each tracker has its own associated clipping region // initialize 1st motion tracker tracker =new MotionTracker(webcam,0,0,200,200); // initialize 2nd motion tracker tracker2 =new MotionTracker(webcam,300,300,500,500); SCM-CityU

7. Class Exercise • Extend the previous example to have two clipping regions as follows • Hint: createtwo trackers with their own clipping regions • How trackers work at the overlapped areas of clipping regions? SCM-CityU

8. Example: Ping-Pong Game • MotionTracker-PingPong-Skeleton.fla in W drive ball box2 barL barR box SCM-CityU

9. Class Exercise • Use trackerand tracker2to control barLand barR, respectively SCM-CityU

10. Class Exercise • Make ballbounce against stage borders • Please refer to Ping-Pong example in Topic 05 SCM-CityU

11. Hit Test between Ball and Bars • Let’s just use hitTestObjectmethod if(ball.hitTestObject(barR)){ trace("barR got hit"); } if(ball.hitTestObject(barL)){ trace("barL got hit"); } SCM-CityU

12. Hit Test between Ball and Bars • Collision response Case 2: Ball is on right sideof barR Case 1: Ball is on left sideof barR SCM-CityU

13. Hit Test between Ball and Bars • But how to determine which case to handle? • Solution: use signs of ball’s speedX • speedX > 0  Case 1 • Response: put ball exactly at the leftsideof barR • speedX < 0  Case 2 • Response: put ball exactly at the rightsideof barR ball.x = barR.x - barR.width * 0.5 - r SCM-CityU

14. Class Exercise • Add sound effects when ball hits stage borders/bars • You can use Flash built-in sounds • Hints • Export sounds forActionScript var ws:WoodSound =new WoodSound(); ws.play(); SCM-CityU

15. Moving Speed • Use MotionTracker’s speedXand speedYto get the moving speed of tracked object • speedX (in pixels) • Positivevalue if object moves along positive x-axis • Negativevalue if objects moves along negative x-axis • speedY (in pixels) • Positivevalue if object moves along positive y-axis • Negativevalue if objects moves along negative y-axis trace(tracker.speedX, tracker.speedY); SCM-CityU

16. Example • Transfer moving speed of tracked object to ball • E.g., hand • Demo SCM-CityU

17. Example • Let’s modify the previous ping-pong example • Keep one tracker only, without any clipping region • No vertical bars • Add friction SCM-CityU

18. Example • Change ball’s speeds only if there is some intersection between ball and moving object if(tracker.isTracked()&& tracker.hitTestPoint(ball.x, ball.y,true)){ ball_speed_x += tracker.speedX; ball_speed_y += tracker.speedY; } SCM-CityU

19. Summary of MotionTracker • Specify source path pointing to the library SCM-CityU

20. Summary of MotionTracker • Withoutclipping region • Withclipping region • cl, ct, cr, cb are with respect to top-left cornerof webcam tracker =newMotionTracker(webcam); tracker =newMotionTracker(webcam, cl,ct,cr,cb); SCM-CityU

21. Summary of MotionTracker • Properties • trackX, trackY: center of tracked region • With respect to top-left corner of webcam • speedX, speedY • Moving speed of tracked object (0, 0) (trackX, trackY) Spring 2010 21 21 SCM-CityU

22. Summary of MotionTracker • Methods • hitTestObject • To test if bounding boxes of display object and tracked region intersect with each other • hitTestPoint • To test if a given point is within the tracked region • isTracked • To check if any moving object is tracked • hideTrackedArea • hideTrackedCenter SCM-CityU

23. Motion-Driven Buttons SCM-CityU

24. Button Click • Usually done via mouse click SCM-CityU

25. Button Click • Can we click digital buttons by our fingers directly? • Answer: Yes! • Through motion • Of course, you canalso rely on touchsensors • Need extra hardware SCM-CityU

26. Today’s Examples • Motion-driven button interaction SCM-CityU

27. Motion Tracking Library • In cityu.scm package (developed by Hongbo Fu) • Webcam class • An easy-to-use wrapper for webcam access • MotionTracker class • Main class for motion tracking • MotionTrackerButtonclass • Use motion to trigger button clicking SCM-CityU

28. Key Idea • Analyze motion over a button on the stage • If motion is detected over it, dispatch OVER event • Cf. MouseEvent.MOUSE_OVER • If motion continuously exists for a while, dispatch CLICKED event • Cf. MouseEvent.MOUSE_DOWN • To avoid undesired clicks after dispatching CLICKED event, clicking is disabled for a short period (say 2 sec) • If clicking is re-enabled, dispatch CLICKABLE event • Cf. MouseEvent.MOUSE_UP SCM-CityU

29. MotionTrackerButton Class • To create a motion-driven button, you need • Step 1: create some display object (e.g., btn) on stage • By either design tool or ActionScript programming • Step 2: create a MotionTrackerButton object and associate it with btn • Step 3: add event listeners for motionBtn • MotionTrackerButton.OVER MotionTrackerButton.CLICKED MotionTrackerButton.CLICKABLE varmotionBtn:MotionTrackerButton= newMotionTrackerButton(webcam,btn); SCM-CityU

30. Step 1: Create Display Object • MotionTracker-Skeleton.fla in W drive • Draw some shape on stage • Convert it to symbol • Assign an instance name (e.g., btn) SCM-CityU

31. Step 2: Create MotionTrackerButton Object • Before adding a MotionTrackerButton object, we need to initialize a Webcamobject • We don’t need to explicitly create a MotionTrackerobject, since it is automatically generated for the associated button importcityu.scm.Webcam;// import Webcam class varwebcam:Webcam=new Webcam(640,480);webcam.addEventListener(Webcam.READY,onCameraReady); functiononCameraReady(e:Event):void{ addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha=0.4; } SCM-CityU

32. Step 2: Create MotionTrackerButton Object • First import MotionTrackerButtonclass • Then create a button with motion control • 3rd parameter: amount of motion needed to activate clicking • How easy a button can be clicked • 4th parameter: time (in milliseconds) needed to re-enable clicing after dispatching CLICKED event • How oftena button can be clicked importcityu.scm.MotionTrackerButton; varmotionBtn:MotionTrackerButton= newMotionTrackerButton(webcam,btn,10,500); SCM-CityU

33. Step 2: Create MotionTrackerButton Object • Motion detection is clipped to the button area only • To see tracking visualization, make the button a bit transparent SCM-CityU

34. Step 3: Add Event Listeners • Event types • MotionTrackerButton.OVER • MotionTrackerButton.CLICKED • MotionTrackerButton.CLICKABLE motionBtn.addEventListener(MotionTrackerButton.OVER, onButtonOver); motionBtn.addEventListener(MotionTrackerButton.CLICKED, onButtonClick); motionBtn.addEventListener(MotionTrackerButton.CLICKABLE, onButtonClickableAgain); function onButtonClick(e:Event):void{ trace("button clicked"); } SCM-CityU

35. Class Exercise • Task 1: when button is clicked by motion • If btn’s current frame label is “up”, go to frame “down” • If btn’s current frame label is “down”, go to frame “up” • Tips • Add code in onButtonClick • Use currentLabel, gotoAndStop SCM-CityU

36. Class Exercise • Task 2: play with different values for 3rd and 4th parametersof MotionTrackerButton. • 3rd parameter: how easy a button can be clicked • 4th parameter: how often a button can be clicked • Compare • MotionTrackerButton(webcam, btn, 1, 500); • MotionTrackerButton(webcam, btn, 20, 500); • Compare • MotionTrackerButton(webcam, btn, 5, 50); • MotionTrackerButton(webcam, btn, 5, 2000); SCM-CityU

37. Pausing, Resuming and Removing of MotionTrackerButtons • Like MotionTracker’s pause, resume and dispose methods, MotionTrackerButton class has same set of methods for similar functionalities SCM-CityU

38. Pausing, Resuming and Removing of MotionTrackerButtons btnResume.addEventListener(MouseEvent.CLICK, buttonClick); btnPause.addEventListener(MouseEvent.CLICK, buttonClick); btnNextFrame.addEventListener(MouseEvent.CLICK, buttonClick); function buttonClick(e:MouseEvent):void{ if(e.target== btnResume){ motionBtn.resume(); }elseif(e.target== btnPause){ motionBtn.pause(); }elseif(e.target== btnNextFrame){ motionBtn.dispose(); removeChild(webcam); webcam.dispose(); gotoAndStop(2); } } SCM-CityU

39. Example: Piano with Motion Control • How to interact multiple MotionTrackerButton objects? SCM-CityU

40. Music Notes • Like previous digital piano example, we use an array to store music notes var notes:Array=[new S1(),new S2(), new S3(),new S4(), new S5(),new S6(), new S7()]; SCM-CityU

41. Create Motion Buttons • We use arrays to store both piano keys and their corresponding motion buttons • Corresponding key and motion button have same index // a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array=newArray(); var keys:Array=[key1, key2, key3, key4, key5, key6, key7]; for(var i:uint =0; i < keys.length; i++){ keysMB.push(newMotionTrackerButton(webcam,keys[i],5,500)); } SCM-CityU

42. Event Listeners for(var i:uint =0; i < keys.length; i++){ // motion-driven keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed); keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased); } functiononKeyPressed(e:Event):void{ for(var i:uint =0; i < keys.length; i++){ if(keysMB[i]== e.target){ trace("index of pressed key", i); } } } SCM-CityU

43. Extension to Other Instruments SCM-CityU