An Experimental System for the Collaborative Control of Unmanned Air Vehicles

1. An Experimental System for the Collaborative Control of Unmanned Air Vehicles Raja Sengupta, CEE Systems, UC Berkeley Joint work with Karl Hedrick, ME UC Berkeley Graduate Students- Tim McGee, Elaine Shaw, Xiao Xiao, Jack Tisdale, Dan Coatta, David Nguyen, Allison Ryan, Mark Godwin, Sivakumar Rathinam , Marco Zennaro, John Cason, and Dan Prull Post Docs- Derek Caveney, Zu Kim, Stephen Spry Engineers- Aram Soghikian, Susan Dickey, Dave Nelson

2. Collaboration Research Goals • Study distributedmechanisms for the collaboration of Unmanned Vehicles • Air to air communication used • Generalize a large number of missions under one framework • Surveillance/Mapping • Border Patrol • Search & Rescue • Convoy Protection, etc. • Create a system that: • Can accommodate a large number of agents (UVs) • Displays tolerance to communcation, hardware failure faults • Capable of running in real time

3. In Action

4. Mission Control

5. Commander View

6. Wing-Mounted Camera allowing for vision-based control, surveillance, and obstacle avoidance Ground-to-Air UHF Antenna for ground operator interface GPS Antenna for navigation 802.11b Antenna for A-2-A comm. Payload Tray for on-board computations and devices Payload Switch Access Door for enabling / disabling on-board devices Current UAV Platform Configuration

7. Off-the-shelf PC-104 with custom Vibration Isolation Orinoco 802.11b Card and Amplifier for A-2-A comm. Analog Video Transmitter for surveillance purposes Printed Circuit Board for Power and Signal Distribution among devices. Umbilical Cord Mass Disconnect for single point attachment of electronics to aircraft. Keyboard, Mouse, Monitor Mass Disconnect for access to PC-104 through trap door while on the ground. Current Payload Configuration

8. MLB Bat IVAircraft • Improved payload weight (25lbs) and volume • Improved logistics: 7.5 hour duration, onboard generator

9. Autopilots and Sensors • Piccolo II autopilots • 4Hz GPS updates (compared to 1 Hz in the old system) • Improved gyros, leading to much better attitude estimation • Analog I/O ports, allowing integration of user-specified sensor inputs into the core autopilot structure • Satellite communication capability • Sensors • several new types: IR camera, radar, IMU, gimbals • will allow expansion of efforts in mapping, vision-based tracking, and control based on other sensor types. • will allow testing and comparison of the effectiveness of various sensors for particular tasks • will allow exploration of how sensor types in a heterogeneous UAV team can be used together in a complementary way.

10. Communications and Video Link • New air-to-air communications system • amplified 802.11b • testing of collaborative team control concepts using short-range air-air comm. • will retain long-range, low-bandwidth, air-ground links • New video downlink system • better monitoring of aircraft video streams • will allow ground-based testing of image processing algorithms and human-machine interface systems. Ground Station

11. Future Experimental System • DURIP funded multi-aircraft testbed • Six Primary Components • A. Five new aircraft with improved payload capacity and configuration • B. Upgraded autopilots with improved autopilot functions • C. New sensors-(bullet cameras, fisheye lenses, IR camera, radar, IMU, gimbals) • D. New air-to-air communications system • E. New video downlink system • F. Trailer/operations center

12. System Architecture UAV UAV UAV Mission Control missions Mission to task decomposer Team Level Task allocation/ Conflict Resolution

13. BLCC- Berkeley Language for Collaborative Control • Define the mission and communicate it to team members • Define the “state” of each agent • Define the mission “state” • Allow for faults • Allow for conflict resolution • Define the information to be communicated between agents.

14. Current Collaborative Architecture Tasking Conflict Allocated Task 1 Conflict Resolution Reallocated To Task 2 Task 1 Border Patrol Allocated Task 1 Task 2 Location Visit

15. Example Scenario subtasks start indicates comm. obstacles UAVi fault

16. Mission State • Each agent communicates primarily through a list of tasks that is shared between UVs. • A task is often described by a location, such as a GPS position. 1 2 3 4

17. Current Collaborative Architecture • Mission statements are decomposed into tasks and relayed from the ground to all aircraft. • Each plane without a task picks the closest available task for itself. Each plane allocates tasks only for itself. • Conflicts in task allocation are resolved using Euclidean distance. • Each aircraft broadcasts its current state and its knowledge of other vehicles’ states. It only has overwriting permissions for its own state. • Current objective • If the airplanes communicate sufficiently often each task will eventually be done • More involved communication, tasking, and conflict resolution protocols are currently under development for future system integration

18. Simulation Results

19. Aircraft Level Architecture Aircraft Avionics Ground Commands Payload Piccolo Camera Database Vision Process. Orinoco Comm. A-2-A Switchboard Orbit Control Waypt Control Vision Control Payload Responsible for Relaying Commands between the PC-104 and Mission Control Piccolo Responsible for Relaying Commands between the PC-104 and Aircraft Avionics Database Permits inter-process communication Vision Processing Frame Grabbing Capabilities Orinoco Inter-vehicular communication protocol Switchboard Task Allocation, Conflict Resolution, and Controller Switching Orbit Control For Closed-Loop Multi-Point Paths Waypoint Control For Single-Point Visits Vision Control For Turn-Rate Based Path Following

20. Generalization: Vision Based Following of Locally Linear Structures(Closed Loop on the California Aqueduct, June 2005)

21. Results – Canal Following • The road detection algorithm runs at 5 Hz (takes < 200 ms) or faster on the PC104 (700 MHz, Intel Pentium III). • No visible error was found from video sequences of over 100 frames containing the canal

22. Cal Road Detection on MLB Video(No Control) • Generic corridor • detection by one- • dimensional • learning • Roads • Aqueducts • Perimeters • Pipelines • Power Lines

23. Vision Based Obstacle Avoidance System

24. Conclusion • Built an unmanned air vehicle system for experimental work on collaboration • Currently four airplanes • Five more planned • Current missions • Visit location and send picture • Border patrol • GPS based • Vision based • Collaboration • Mission to task decomposition • Each mission should have its own semantics of decomposition • Autonomous in-air task division and conflict resolution • Currently limited to static tasks

25. Geographic Data Management

26. Scalable Information Management:Target Map and Risk Map Risk Map Example: Target Map • Target distribution map • P(A, N, t); probability of N targets of type t in area A • Target distribution update • Fuses measurements from different kinds of sensors (SAR and EO) • Bayesian update • Risk map computation • Integral of threat model with respect to the measure P(A, N, t) • Generates the value function for navigation UCB Rathinam 2003

27. Scalable Information Management:Distributing the Publisher Service Voronoi tessellation Data objects Euclidean Space Publishing Servers • Geographic Data Management Network Sengupta AINS 2003

28. Scalable Information Management:Distributing the Publisher Service desired data delivery server Metric Space User

29. Movie of Implementation Total data is this map • 4 laptops over wireless • One publisher per laptop • Start with one publisher • Three others come up • Some die • Data redistributes as publishers join and leave

30. Movie of Implementation Total data made of many data objects • 4 laptops over wireless • One publisher per laptop • Start with one publisher • Three others come up • Some die • Data redistributes as publishers join and leave

31. Movie of Implementation Voronoi tessellation • 4 laptops over wireless • One publisher per laptop • Start with one publisher • Three others come up • Some die • Data redistributes as publishers join and leave

32. Movie of Implementation • 4 laptops over wireless • One publisher per laptop • Start with one publisher • Three others come up • Some die • Data redistributes as publishers join and leave

33. Movie of Implementation • 4 laptops over wireless • One publisher per laptop • Start with one publisher • Three others come up • Some die • Data redistributes as publishers join and leave

34. Scalable Information Management:Distributing the Publisher Service Movie of our Implementation 4 servers on 4 laptops over wireless

35. Data Consistency in the Publisher:Inconsistent copies are detected whp Wrong location copy 1 Data Location Wrong location copy 2

36. Data Consistency in the Publisher:Drift in a 2-D Markov Process

37. Geographic Data Management Network:Survivable Information for UAV Swarms • The server backbone dynamically tracks the client agent organization • Servers move in and out while the information survives

38. Tracking the Agent Organization:Dynamic GDMN backbone Control • Design a distributed control algorithm for the servers to partition the data and the clients to minimize the total bit-meters (Kumar etal.) of work done in the system and balance the load on the servers. • Let the load generated in each client be bi. If the locations of the points are denoted by pi and the location of the servers are denoted by cj, then the total cost is:  bi ( min dist(pi, cj) )  i  j • The control algorithm updates server positions to reduce this cost

39. Simulation • This example involves 100 clients and 6 servers

40. Control algorithm • In each sampling interval, each server • Measures the positions and the traffic generated by its clients • GDML routing protocols make the client set the Voronoi cell • Calculates the weighted centroid of all the clients it serves • Moves towards its weighted centroid • Works well if the servers travel faster than the clients • The algorithm is based on the k-means algorithm (MacQueen ,1967 )

41. Publications: 2005

42. Publications: 2005 cont’d

43. Publications: 2005 cont’d

44. Publications: 2004

45. The End

46. Publications: 2004 cont’d

47. Publications: 2003

48. Theses and Dissertations

49. The End