Control of UAV’s

1. Control of UAV’s Raja Sengupta (sengupta@ce.berkeley.edu) Assistant Professor Civil and Environmental Engineering: Systems UC Berkeley Joint Work with the C3UV team

2. Challenge of Flying Low • Helicopter pilots fly low • FAA requires see and avoid • Find the freeway and follow it

3. Used Sectionals to build a Manhattan model at 300 feet (approx.) • Simulation testing of Control

4. Flying Low: Strategy • Helicopter pilots fly low • Find the freeway or waterway and follow it • Avoid few remaining obstacles

5. Cal Freeway Detection on MLB Video(No Control) • Generic corridor • detection by one- • dimensional • learning • Roads • Aqueducts • Perimeters • Pipelines • Power Lines

6. Road Following:Closed Loop Control, August 2003

7. Generalization: Vision Based Following of Locally Linear Structures(Closed Loop on the California Aqueduct, June 2005)

8. Obstacle Avoidance

9. Tailored to……………….. • For most UAV applications (>50 m), the obstacles of concern will be large objects such as towers, buildings or large trees • For these cases, the problem of obstacle detection is different from that of ground vehicles in environments cluttered with many obstacles. VS

10. Flight Demonstration • Experiment flown on a Sig Rascal airframe with a Piccolo avionics package and vision processing on an onboard PC104. • An 8.5 foot diameter balloon was used as the obstacle (distance currently calculated using GPS).

11. Flight Demonstration

12. Flight Demonstration -100 -100 0 0 100 100 200 200 300 300 400 400 0 0 Balloon Balloon avoidance with GPS avoidance with GPS -50 -50 -100 -100 -150 -150 direction of flight direction of flight -200 -200 y position (m) y position (m) -250 -250 -300 -300 -350 -350 -400 -400 autonomous control autonomous control started started -450 -450 -500 -500 x position (m) x position (m)

13. Distributed Information Management for Collaboration

14. Objective: Distribute the data objects shared by the team across the members of the team UAV UAV Team publisher Operation decomposer Team Level Resource allocation/scheduling Operation monitor Dispatcher Formation Navigation UCB Rathinam 2004

15. Scalable Information Management: Voronoi tessellation Data objects Euclidean Space UAV’s • Geographic Data Management Network Sengupta AINS 2003

16. Scalable Information Management: desired data request delivery agent Metric Space

17. Distributed Implementation in Action Movie of our Implementation 4 agents on 4 laptops over a wireless LAN

18. End

19. Detailed Slides

20. Results Tracking the California Aqueduct • The average error of the position of the vehicle from the curve was 10 meters over a length of 700 meters of the canal.

21. Results – Canal Following • The road detection algorithm runs at 5 Hz (takes < 200 ms) or faster on the PC104 (700 MHz, Intel Pentium III). • No visible error was found from video sequences of over 100 frames containing the canal

22. Flying Low • Helicopter pilots fly low • Find the freeway and follow it • FAA requires see and avoid • obstacles and aircraft

23. Obstacle Avoidance is a Constraint: Not a MissionApproach: Safe Set-theoretic • Assume capabilities of the airplane • Compute an unsafe set • When in the safe set, execute the mission • On the boundary execute the obstacle avoidance control • Assume obstacles are • Sparse • Stationary • Rectangular

24. As the UAV avoids an obstacle it slides on the boundary of the unsafe set

25. Analytical Solution Pl • The analytical solution can be calculated in 5 ms Cr_ns Cr Oa Ocusp y (m) Ona BRS Cl Pr x (m)

26. Cal UAV: Target CapabilitiesObstacle Avoidance • Simulation testing of Control • Flight through Manhattan model (300+ feet)

27. Related Work • Vision-based obstacle avoidance has been studied primarily in the context of mobile ground robots. • Lenser ’03, Ohya ’00, Lorigo ‘97, • Vision based navigation of UAVs • Saripalli ’02, Shakernia ’02, Furst ’98 – Landing with known markings • Sinopoli ’01, Doherty ‘00 – Visual landmark navigation (terrain avoidance) for helicopter • Ettinger ’02, Pipitone ’01, Kim ’03 – Pose estimation for aircraft • Obstacle/Collision Avoidance for UAVs • Mitchell ‘01 – Aircraft avoiding known aircraft • Sigurd ’03 – Aircraft with magnetic sensors • Sastry ‘03 – Helicopters avoiding known helicopters/obstacles • How ’02 – MILP for Obstacle Avoidance • Vision based obstacle avoidance • Barrows ’03 – Biomimetic reactive control

28. Related Research • Ground robots • Fixed baseline stereo – JPL, many others • Monocular map construction – Lenser (CMU), Kim (Berkeley) • Cooperative stereo - CMU • Optical Flow • Helicopter ground following – Srinivasan/Chahl (Australia) • Corridor following - USC helicopter • Micro UAV obstacle avoidance – Centeye • UAV depth map construction • Lidar – CMU Helicopter Project, Sastry (Berkeley Helicopter Project). • Vision + high precision IMU – Bhanu (joint with Honeywell) • Stereo Vision • GT Helicopter

29. Requires DepthTypically use Stereo Vision • Given the image coordinates of a feature in one image • if one can find the image coordinates of the feature in the other image (feature matching), and • if one knows the rotation and translation of the two image planes then one knows the world coordinates of the feature (Ego-motion Estimation)

30. Problem with Depth Estimation by Stereo Vision 0 Z+ Z Z- z Increased accuracy requires increased camera separation

31. Accurate Depth Estimation is a Problem • Range error due to pixel errors is .

32. Approach • UAVs flying at low altitudes must autonomously avoid obstacles • Strategy • Segment the image into sky and non-sky • Non-sky in the middle  OBSTACLE • Strategy 1 • Aim at the sky • Strategy 2 • If it looms faster than a threshold and is in the middle  AVOID Else do NOTHING

33. Segmentation at Moffet Airfield • Results for multiple regions found (only largest regions shown, dark blue represents all small regions)

34. Sky Segmentation

35. Vision Processing • Classification: balloon/horizon correctly found in ~ 90% of images • Time results: ~2Hz (120ms SVM, 200-600 ms horizon)

36. Obstacle Avoidance: Next Steps • Loom: 4 pixels/second asuming a 70deg FOV camera with 320 pixels, Speed :20 m/s • turn radius 100 m, processing delay of 0.5 s, safe avoidance distance of 10 m, the minimum obstacle size is about 2.5 m

37. Theoretical Work and Tool Development

38. Resource Allocation Algorithms for Multi-vehicle Systems with Nonholonomic Constraints

39. Vehicle Target Path Planning Problem • Vehicles: V = {v1,v2,...vn}, Targets:{t1,t2...tm}, Angles of approach: {θ1, θ2... θm} • Assign a cycle Pi for each vehicle starting and ending at vertex vi such that Ui Pi=V. A cycle Pi is a ordered sequence of vertices {ti1,...tik} for the vehicle i to visit • Assign paths for each vehicle that satisfy the non-holonomy constraints for the given sequence. • The objective is to minimize ∑Cost(Pi)

40. Resource Allocation or Vehicle-Target Assignment • Given a collection of targets that need to be serviced and a collection of vehicles, how do you assign vehicles to targets ? • 1-1 Vehicle Target Assignment • Vehicle-Target Path Planning • More targets than vehicles

41. 1-1 Vehicle Target Assignment Objective: Constraints: • Solution to the relaxed linear programming formulation is a feasible solution for the assignment problem • Total unimodurality

42. Vehicle Target Path Planning Problem • Traveling salesman problem if number of vehicles = 1 and no kinematic constraints • Asymmetry, that is dij ≠ dji but satisfies triangular inequality • c-Approximation algorithm: cost of the solution is atmost c times the optimal value • Approximation algorithms for asymmetric TSP • Based on the number of points visited: 0.99log(n) - Markus Blaser (2002) • Based on the ratio of the distances dmax/dmin - Kumar and Li (2002) • Single vehicle problem with non-holonomy - Bullo et al, 2005 • Multi vehicle problem with heuristics – Zhijun et al, 2005 • Asymmetry with kinematic constraints can be bounded if the euclidean distance between the points are reasonably apart, d ≥ 2R • Sensor footprints are at least of the order of the minimum turning radius. • Basic idea: • Solve the problem assuming the distances are Euclidean • Using the sequence for each vehicle, assign paths that satisfy kinematic constraints

43. Motion Planning • Find the minimal distance path joining ( x1,y1 ,1) and ( x2,y2,2) subject to kinematic constraints

44. Algorithm for Vehicle Target Path Planning Problem 0 0 0

45. Algorithm for Vehicle Target Path Planning Problem 0 0

46. Algorithm for Vehicle Target Path Planning Problem Find the Eulerian walk for each subtree and reduce it to a TSP tour for each vehicle 2 approximation algorithm because the cost of the multigraph is ≤ 2*Cost(MST)

47. Algorithm for Vehicle Target Path Planning Problem

48. Algorithm for Vehicle Target Path Planning Problem

49. Algorithm for Vehicle Target Path Planning Problem • Theorem: The approximation algorithm has a bound of = 2ddubins/deuclidean ≈ 6 • Further the distance between the points, better the bound is • At best it could be ≈ 2 • Angles of approach for the targets were given A B

50. Conclusions • Addressed the problem of resource allocation in the context of unmanned aerial vehicles • Algorithms for multiple vehicles satisfying kinematic and fuel constraints. • Tighter bounds for the algorithms • Future work could address vehicles with fuel constraints, targets with precedence constraints etc.