Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs)

1. Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Mitul Saha and Pekka Isto Artificial Intelligence Lab Stanford University Research Institute for Technology University of Vaasa, Finland Research supported by NSF

2. There has not been much development in manipulation planning for deformable objects because • it is difficult to model and predict the deforming nature of • deformable objects • struggle in basic motion planning Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

3. There has not been much development in manipulation planning for deformable objects because • it is difficult to model and predict the deforming nature of • deformable objects • struggle in basic motion planning Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

4. There has not been much development in manipulation planning for deformable objects because • it is difficult to model and predict the deforming nature of • deformable objects • struggle in basic motion planning Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

5. There has not been much development in manipulation planning for deformable objects because • it is difficult to model and predict the deforming nature of • deformable objects • struggle in basic motion planning Manipulation Planning Research so far… • The ability to autonomously manipulate objects is one of most desirable features in a robot. Hence manipulation planning has been an active area of research for the last many decades • So far, manipulation planning research has mainly focused on manipulating rigid objects • We have been interested in manipulation planning for deformable objects, because a large number of objects that we handle in our daily lives are deformable to some extent

6. sailing knot figure-8 knot bowline knot Manipulation Planning for Deformable Linear Objects (DLOs) GOAL: to develop a motion planner that would enable robots to autonomously manipulate Deformable Linear Objects (ropes, cables, sutures) in various settings. knot tying in daily/recreational life laying/loading cables in industrial settings suturing in medical surgery robot dress autonomous robotic DLO manipulation

7. Manipulation Planning for Deformable Linear Objects (DLOs) Challenging • The DLO manipulation problem is extremely challenging for robotics because • being highly deformable, they can exhibit a much greater diversity of behaviors, which are hard to model and predict • identifying topological states of DLOs is coupled with some unsolved problems in knot-theory/ mathematics Interesting • The DLO manipulation problem has a nice structure. • It brings together robotics, knot theory, and computational • mechanics.

8. Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. • - knot simplified using Reidemeister moves (RM) • from knot theory • one robot used to execute the RMs • assumes DLO resting on a plane

9. Previous Related Work “Planning of One-Handed Knotting/Raveling Manipulation of Linear Objects”, IEEE ICRA 2004, Wakamatsu, et. al. • - knot simplified using Reidemeister moves (RM) • from knot theory • one robot used to execute the RMs • assumes DLO resting on a plane Our contribution: -DLO need not be in a plane -We use more than one robot in coordination -We consider collision constraints (robot-DLO, robot-obstacle) -We consider the physical behavior of the DLO while planning -We consider interaction of the DLO with other objects

10. The Manipulation Problem available robot arms How do we define goal configurations?

11. Defining Goal Configurations • Goal configurations are defined in terms of topology instead of exact geometry Geometrically different but topologically same: Bowline knot while winding, number of wounds more important

12. Defining Goal Configurations • In knot theory, crossing configuration of a curve is used to characterize its topology planar projection of the DLO central axis

13. make them part the DLO how to account for interactions with other objects? semi-deformable linear object (sDLO) Defining Goal Configurations • In knot theory, crossing configuration of a curve is used to characterize its topology crossing: local self-intersections C2: (-2,5)- planar projection of the DLO central axis C3: (3,-8)- C1: (1,-6)- sign of a crossing C4: (-4,7)- Crossing Configuration: (C1, C2, C3, C4): ((1,-6)-, (-2,5)-, (3,-8)-, (-4,7)-)

14. Recent successes in computational mechanics: Elastic thread model: [Wang, et al., 05] Nylon thread model: [Dhanik, 05] Suture model: [Brown, et al., 04] Physical modeling of the DLO We take as input the physical model of the DLO in the form of a state transition function f:

15. Manipulation Tools • Manipulation using 2 cooperating robot arms

16. Manipulation Tools • Manipulation using 2 cooperating robot arms • Use of static sliding supports (“tri-needles”) to provide structural support

17. Basis of our Planning Approach • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time Forming Sequence: C2, C1, C4, C3

18. Basis of our Planning Approach C2 C3 • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time C1 C4 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence” Forming Sequence: C2, C1, C4, C3

19. Basis of our Planning Approach C2 C3 • Defining “Forming Sequence” walk along the DLO; crossing “formed” when encountered the second time C1 C4 A DLO topology or knot can be tied, crossing-by-crossing, in the order defined by its “forming sequence” Forming Sequence: C2, C1, C4, C3 • Defining “loop hierarchy” used to determine the placementof static sliding supports (“tri-needles”)

20. forbidden region search tree Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search • use theloop hierarchyto placestatic sliding supports (tri-needles)

21. forbidden region search tree Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search • use theloop hierarchyto placestatic sliding supports (tri-needles) Robot A Robot A grasping robot fails DLO Robot B

22. forbidden region search tree Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search • use theloop hierarchyto placestatic sliding supports (tri-needles)

23. forbidden region search tree Our Manipulation Planning Algorithm -search the configuration-space using a sampling-based tree -use physical model to sample new DLO shapes -use forming sequence to bias search • use theloop hierarchyto placestatic sliding supports (tri-needles) loop hierarchy tri-needles

24. sailing knot bowline knot Results • -Planner implemented in C++ • -Took 15-20 minutes on a 1GB, 1GHz processor to generate manipulation plans for tying popular knots: bowline, neck-tie, bow (shoe-lace), and stunsail • Videos: • http://ai.stanford.edu/~mitul/dlo neck-tie bow

25. Results

26. Results neck-tie

27. bowline knot robustness dues to tri-needles Results In the real-life, we have tested the ability of the planner to generate robust plans by tying the popular Bowline knot with various household ropes on a hardware platform with two PUMA robots, using the manipulation plan generated by the planner.

28. Future Plans collaboration with General Motors suturing in medical surgery Conclusion • We have developed a motion planner for manipulating deformable linear objects (such as ropes, cables, sutures) in 3D using cooperating robots. - it can tie self-knots and knots around rigid objects - unlike in traditional motion planning, goals are topological and not geometric - we account for the physical behavior of the DLO - it is robust to imperfections in the physical model of the DLO - it is first of its kind (we not aware of any other planner for computing collision-free robot motions to manipulate a DLO in environments with obstacles) - the implemented planner has been tested both in graphic simulation and in real-life on a dual-PUMA-560 hardware platform

29. Motion Planning for Robotic Manipulation of Deformable Linear Objects (DLOs) Acknowledgement: Advisory: Jean-Claude Latombe PUMA experiments: Oussama Khatib, Irena, Jaehueng Park, Jin Sung Physical models of ropes: Etienne Burdet, Wang Fei (EPFL) Useful comments: anonymous reviewers

30. - Tight knots Semi-tight knots We focus on two types of common knots: under over over Crossing Configuration: ((1,-6)-, (-2,5)-, (3,-8)-, (-4,7)-)

31. Needle Placement