1 / 49

Towards Practical Visual Servoing in Robotics

Towards Practical Visual Servoing in Robotics. R. Tatsambon Fomena. Example of a f ully integrated system. Manus ARM, iARM  Exact Dynamics Joysticks and keypads common user interfaces. Image from http://www.exactdynamics.nl. Examples of a fully integrated system.

natala
Download Presentation

Towards Practical Visual Servoing in Robotics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Towards Practical Visual Servoing in Robotics R. TatsambonFomena

  2. Example of a fully integrated system ManusARM, iARM Exact Dynamics Joysticks and keypadscommonuser interfaces Image fromhttp://www.exactdynamics.nl

  3. Examples of a fully integrated system http://www.youtube.com/watch?feature=player_embedded&v=LBUyiaAPCcY Click-graspapplicationwiththeintelligentManusArm

  4. Visual servoing: Example of a fully integrated system 1. User selects object The position of the object is computed using stereo vision from the shoulder of the camera 2. Robot arm moves to that position expressed in its base frame 3. Having the object in view the robot arm computes a more precise target and adjust orientation 4. Using left gripper camera, robot searches database for best object match. 5. Using the object template the robot arm moves to align the feature points . 6. Once aligned gripper moves forward and closes its gripper. 7. Robot returns object to user. (Tsuiet al., JABB, 2011)

  5. Kinova Robotics aims also for a similar system Joystick control.3 Control Modes: • Move robot's hand in three dimensional space, while maintain the orientation of the hand. • User can modify orientation of the hand, but keeping hand centered at the same point in space. • User can grasp and release of the hand using either two or three fingers. http://www.youtube.com/watch?feature=player_embedded&v=O0nr8NdV6-M http://www.youtube.com/watch?feature=player_embedded&v=vV4tbS7WTL0 Image from http://kinovarobotics.com

  6. Visual servoing: The control concept Robot + Camera(s) World HRI Specification of Goal S* S* ACTION - + S PERCEPTION perception for action • (Espiauet al., TRA, 92) (Hutchinson et al., TRA, 96)

  7. Visual servoing: Why visual sensing? • How to control the position of the end-effector of a robot with respect to an object of unknown location in the robot base frame? • How to track a moving target? A visual sensor provides relative position information

  8. Visual servoing: How can you use visual data in control? • Look then move • Visual feedback control loop Robot + Camera(s) Robot + Camera(s) PERCEPTION S-S* ACTION S* ACTION S PERCEPTION - +

  9. Quiz What are the advantages of closed-loop control over open loop control approach?

  10. Visual servoing: Ingredients for a fully integrated system • HRI • Visual tracking method • Motion control algorithm Robot + Camera(s) S* ACTION S HRI Specification of Goal S* PERCEPTION

  11. Visual servoing: Visual tracking • Crucial as it provides the necessary visual feedback • coordinates of image points or lines • Should give reliable and accurate target position in the image PERCEPTION S S=(x,y) Selection of the set of Measurements to use for control Camshift color tracker provides 2D (x,y) coordinates of the tracked objects Current image Tracker searches for the end-effector

  12. Visual Tracking Applications: Watching a moving target • Camera + computer can determine how things move in an scene over time. Uses: • Security: e.g. monitoring people moving in a subway station or store • Measurement: Speed, alert on colliding trajectories etc.

  13. Visual Tracking Applications: Human-Computer Interfaces • Camera + computer tracks motions of human user and interprets this in an on-line interaction. • Can interact with menus, buttons and e.g. drawing programs using hand movements as mouse movements, and gestures as clicking • Furthermore, can interpret physical interactions

  14. Visual Tracking Applications: Human-Machine Interfaces • Camera + computer tracks motions of human user, interprets this and machine/robot carries out task. • Remote manipulation • Service robotics for the handicapped and elderly

  15. Visual servoing: Example of visual tracking Registration based Tracking Nearest Neighbor tracker N vs Efficient Second Order Minimization http://www.youtube.com/watch?v=do5EQGMpv50

  16. Visual servoing: Motion control algorithm • 3 possible control methods depending on the selection of S: 2D, 3D, and 2 ½ D (Corke, PhD, 94) 3D VS 2 ½ D VS 2D VS High bandwidth requires precise calibration: camera and robot-camera

  17. Visual servoing: Motion control algorithm • Key element is the model of the system (Corke, PhD, 94) 3D VS 2 ½ D VS 2D VS 2D VS 2 ½ D VS 3D VS Abstraction for control Robustness to image noise, calibration errors Suitable for unstructured environments

  18. 3D Visual servoing (Wilson et al., TRA, 1996) How to sense position and orientation of an object?

  19. 2-1/2 D = Homography-based Visual servoing (Maliset al., TRA, 1999) Euclidean Homography?

  20. 2D Visual servoing (Espiauet al., TRA, 1992) (Jagersandet al., ICRA, 1997) Example of 2D features? http://www.youtube.com/watch?v=Np1XFuDFcXc

  21. Quiz What are the pro and cons of each approach? 1-a) 2D 1-b) 3D 1-c) 2 ½ D

  22. Visual servoing: Motion control algorithm • Key element is the model of the system: how does the image measurements S change with respect to changes in robot configuration q? can be seen as a sensitivity matrix

  23. Visual servoing: Motion control algorithm • How to obtain ? 1) Machine learning technique • Estimation using numerical methods, for example Broyen 2) Model-based approach • Analytical expression using the robot and the camera projection model • Example S=(x,y) (Jagersandet al., ICRA, 1997) How to derive the Jacobian or interaction matrix L?

  24. Visual servoing: Motion stability • How to move the robot knowing e = S-S* and ? • Classical approach: the control law imposes an exponential decay of the error Classical control

  25. Visual servoing: Motion control algorithm :=VisualTracker(InitImage) Init = Init While ( > T ) { CurrentImage := GrabImage(camera) := VisualTracker(CurrentImage) Compute = Estimate Compute Change robot configuration with }

  26. Visual servoing: HRI • Important for task specification • point to point alignment for gross motions • points to line alignment for fine motions • Should be easy and intuitive • Is user dependent

  27. (Hager, TRA, 1997)

  28. (Kragic and Christensen, 2002) How does the error function looks like?

  29. (Kragic and Christensen, 2002) How does the error function looks like?

  30. (Kragic and Christensen, 2002) How does the error function looks like?

  31. Visual Servoing: HRI-point to point task error • Point to Point task “error”: Why 16 elements?

  32. Visual Servoing: HRI-point to line task error • Point to Line Line: Note: y homogeneous coord.

  33. Visual Servoing: HRI-parallel composition example E (y) = wrench y - y y - y 4 2 7 5 y • (y  y ) y • (y  y ) 8 6 2 4 1 3 (plus e.p. checks)

  34. (Li, PhD thesis, 2013) Maintaining visibility http://www.youtube.com/watch?feature=player_embedded&v=QQJIVh0WICM

  35. Visual Servoing: HRI with virtual visual fixtures Motivation • Virtual fixtures can be used for motion constraints Potential Applications • Improvements on vision-based power lines or pipelines inspection • Flying over a power line or pipeline by keeping a constant yaw angle relative to the line (line tracking from the top) • Hovering over a power pole and moving towards the top of a power pole for a closer inspection https://www.youtube.com/watch?v=5W3HiuOYuhg

  36. Where does virtual fixtures can be useful? • Robot Assistant method for microsurgery (steady hand eye robot) • “Here the extreme challenge of physical scale accentuate the need for dexterity enhancement, but the unstructured nature of the task dictates that the human be directly “in the loop”” - EyeRobot1

  37. Where does virtual fixtures can be useful? • How to assist the surgeon? Cooperative control with the robot • Incorporate virtual fixture to help protect the patient, and eliminate hand’s tremors of the surgeon during surgery -

  38. Where does virtual fixtures can be useful? • Central retinal vein occlusion, solution= retinal vein cannulation • Free hand vein cannulation http://www.youtube.com/watch?v=MiKVFwuFybc&feature=player_embedded • Robot assisted vein cannulation http://www.youtube.com/watch?v=s5c9XuKtJaY&feature=player_embedded • What to prove? “robot can increase success rate of cannulation and increase the time the micropipette is maintained in the retinal vein during infusion”

  39. Virtual fixture: example • JHU for VRCM (Virtual Remote Center of Motion) http://www.youtube.com/watch?v=qQEJEM7YeXY&feature=player_embedded

  40. What is a virtual fixture? • “Like a real fixture, provides surface that confines and/or guides a motion” (Hager, IROS, 2002) Its role is typically to enhance physical dexterity (Bettini et al., TRO, 2004)

  41. What is a virtual fixture? • “Software helper control routine” (A. Hernandez Herdocia, Master thesis, 2012) Line constraint, plane constraint

  42. What is virtual visual fixture? Vision-based motion constrains • Geometric virtual linkage between the sensor and the target, for 1 camera visual servoing(Chaumetteet al., WS, 1994) Extension of the basic kinematic of contacts • Image-based task specification from 2 cameras visual servoing(Doddset al., ICRA, 1999) Task geometric constraint defines a virtual fixture

  43. What is virtual visual fixture? Vision-based motion constraints • Geometric virtual linkage (Chaumetteet al., WS, 1994)

  44. What is virtual visual fixture? Vision-based motion constraints • Image-based task specification (Dodds et al., ICRA, 1999)

  45. Mathematical insight of virtual fixture • As a control law – filtered motion in a preferred direction • As a geometric constraints – virtual linkage • As condition for observer design - persistency of excitation Prove it? • (Hager, IROS, 1997)

  46. Mathematical insight of virtual visual fixture Take home message: Kernel of the Jacobian or interaction matrix in visual servoing (Tatsambon, PhD, 2008)

  47. Summary: where are the next steps to move forward? • The conclusion is clear • So far only a handful existing fully integrated and tested visual servoing system • Mechatronics & theoretical developments than actual practical software development • Our natural environment is complex: Hard to design an adequate representation for robots navigation • It is time to free visual servoing from its restrictions to solving real world problems • Tracking issue: reliability and robustness (light variation, occlusions, …) • HRI problem: Image-based task specification, new sensing modalities should be exploited

  48. Short term research goal: new virtual visual fixture • Virtual fixtures • Line constraint To keep tool on the line Can be done with point-to-line alignment • Ellipse constraint To keep the tool on the mapping of a circle Has never been done

  49. Short term research goal: grasping using visual servoing Instead of having predefined grasping points in a database of objects Where to grasp?

More Related