Wide Area Georeferenced Photography Using Unmanned Aircraft

1. Wide Area Georeferenced Photography Using Unmanned Aircraft Prof. James Lumpp IDEA Laboratory University of Kentucky Electrical and Computer Engineering IDEA Laboratory University of Kentucky

2. Outline • Background • UAV Platforms • Orthorectification and Composite Images • Avionics and Camera Systems • The NASA World Wind UAV plug-in v1.1 IDEA Laboratory University of Kentucky

3. PAX River Competition • Annual Association of Unmanned Vehicle Systems International (AUVSI) Student UAV Competition. • Hosted by the U.S. Navy Program Executive Office for Strike Weapons and Unmanned Aviation • International competition held in July (24 teams in 2006) IDEA Laboratory University of Kentucky

4. The Competition • Takeoff, fly specified GPS waypoints, acquire images, identify and geo-reference ground targets, and land within time limit • Altitude airspace limitations • Dynamic mission requirements • Submit the GPS location and orientation of each target IDEA Laboratory University of Kentucky

5. Develop a UAS Unmanned Aircraft “System” • Airframe and Propulsion • Embedded Computing • Autonomous Flight Control • Communication Links • Image Capture and Analysis • Geo-Referencing photographs IDEA Laboratory University of Kentucky

6. UAS Fleet • Sig Kadet Senior • Wingspan: 80.5 in. • Length: 64.75 in. • Weight: 6.5 lbs unloaded • Engine: 1.6 bhp Tower Hobbies .61 • Sig Rascal 110 • Wingspan: 110 in. • Length: 75.75 in. • Weight: 11 lbs unloaded • Engine: 3.7 bhp OS 1.6FX IDEA Laboratory University of Kentucky

7. Orthorectification • Aerial photographs must be “processed” to remove distortion from the images • Without this processing it is difficult to make direct and accurate measurements of distances, angles, positions, and areas • Errors are introduced by • Topography • Platform induced geometry changes IDEA Laboratory University of Kentucky

8. Airfield Orthorectification IDEA Laboratory University of Kentucky

9. Topography Requires DEM or other techniques IDEA Laboratory University of Kentucky

10. Nadir IDEA Laboratory University of Kentucky

11. Design Team IDEA Laboratory University of Kentucky

12. Autonomous Flight Control (too much text) Key Features of Piccolo System • Avionics - Small size: 4.8” x 2.4” x 1.5”, 233g. • Highly integrated –Includes sensors, GPS, enclosure, and communications link. • Real-Time Data Logging. • Manual piloting over the integrated UHF ground station link. • End-user programmability, including RS-232 serial communications. • Flexible drag and drop flight plan generation and updates. Piccolo Avionics Piccolo System Block Diagram IDEA Laboratory University of Kentucky

13. Tiny Interface Module CAN • Tiny Interface Module implementing CAN networking • Controller Area Network (CAN) • Designed to be tolerant of electromagnetically noisy environments • Initially utilized in automotive applications IDEA Laboratory University of Kentucky

14. TIMCAN Parts JTAG (Microcontroller programming/debug) • Custom harness UART0 (Serial Communications Interface) • Level shifted UART1 • Level shifted and non-shifted SPI (Serial Peripheral Interface) • 4-wire compatible Servo header(x4) • Standard servo plug-in GPIO(x3) (General Purpose I/O) • CAN (x2)(Controller Area Network) • Four pin header • LED (x3)(Light Emitting Diode) • Power • Status 0 • Status 1 • Push Button (x2) • Reset • Generic push button IDEA Laboratory University of Kentucky

15. IDEA Laboratory University of Kentucky

16. TIMWireless • Tiny Interface Module implementing Wireless networking • Wirelessly connected using Xbee modules • Custom API layer built on top of ZigBee communications layer IDEA Laboratory University of Kentucky

17. TIM Wireless 3D View Board Layout IDEA Laboratory University of Kentucky

18. IDEANIX • IDEANIX is a unique build of the embedded operating system microC/OS-II • Its uniqueness comes from being an Intel 8051port which has a set initialization that allows it to run on the Silicon Labs C8051f04x series microcontrollers. IDEA Laboratory University of Kentucky

19. Message Routing Layer (MeRL) Direct Inter-Task Communication Inter-Task Communication with MeRL • MeRL removes the over head of direct Inter-Task Communication • Allows for the seamless integration of inter-processor communication of a broadcast network IDEA Laboratory University of Kentucky

20. MeRL enabled Modular Design • Having a Message Routing Layer which makes inter-processor communication as seamless as inter-task communication allows for a noise/fault tolerant modular design, and makes it ideal for the modular development as such platforms like avionics IDEA Laboratory University of Kentucky

21. Camera Systems • Aiptek IS-DV digital camcorder • Fixed lens camera • NTSC video out • 3MPixels • Built-in SD Card Slot (Max. 1 GB) • TXA5-Rcb 1W-70 cm transmitter @ 426.25MHz • Dual-Axis Stabilizing System IDEA Laboratory University of Kentucky

22. High Resolution Geo-Referencing • EE 499 Senior Design Project • Goals: • To obtain clear, high resolution images of target area • To make geo-referencing data reliable IDEA Laboratory University of Kentucky

23. High Resolution Geo-Referencing • Using a Linux Single-Board Computer to interface with and control the camera via USB. • Libgphoto2 • Library that exists to interface cameras with USB • SBC processes geo-referencing data and matches accurate data with each photo IDEA Laboratory University of Kentucky

24. Photo Stitching System • NASA World Wind • Open source software • GUI to display end images with all required image data • Functionality in a “plug-in” • Images can coexist with NASA World Wind data IDEA Laboratory University of Kentucky

25. Lens distortion terminology • Tangential lens distortion – "decentering", or imperfect centering of the lens components and other manufacturing defects in a compound lens • Radial lens distortion – Distortion present in fisheye lenses • Principal point – The center point of an aerial photograph IDEA Laboratory University of Kentucky

26. "Plumb Bob" model of lens distortion • Contains 5 distortion coefficients (K) • Other inputs – Focal length, principal point, and skew coefficient compose the camera matrix • Not advised to use 5th coefficient, 3rd and 4th are generally not used either • It is claimed that the error in the principal point of today’s cameras is negligible IDEA Laboratory University of Kentucky

27. Matlab camera toolbox • r is radius from principal point to pixel IDEA Laboratory University of Kentucky

28. UAV Plugin 1.1 uses K1 and K2 • Using only the first two is much quicker and results are acceptable • Bilinear interpolation eliminates aliasing effects that cause image to look rough and blocky • All 5 coefficients can be found using the Matlab camera toolbox • OpenCV supports camera calibration IDEA Laboratory University of Kentucky

29. UAV for World Wind Features • Code exists as a plugin (no more compilation required by the developer) • Real-time modification of individual image parameters • Coexists with WW images • Support for pre-processing of images on pixel level (for histograms) IDEA Laboratory University of Kentucky

30. UAV Plugin 1.1 IDEA Laboratory University of Kentucky

31. UAV Plugin 1.1 • Code exists as a plugin (no more compilation required by the developer) • Real-time modification of individual image parameters • Coexists with WW images • Support for pre-processing of images on pixel level (for histograms) IDEA Laboratory University of Kentucky

32. UAV Plugin 1.1 • Uses the first two coefficients of a lens distortion model to quickly correct lens distortion • Bilinear interpolation eliminates aliasing effects that cause image to look rough and blocky IDEA Laboratory University of Kentucky

33. Testing IDEA Laboratory University of Kentucky

34. Competing in Maryland IDEA Laboratory University of Kentucky

35. Internet Vehicle Tracking • Have the ability to remotely enable and disable starting the vehicle using SMS messaging • Vehicle keys are replaced with RFID cards • “Geo-Fencing” using GPS • System will log Vehicle hours for service dates IDEA Laboratory University of Kentucky

36. Internet Vehicle Tracking IDEA Laboratory University of Kentucky

37. Internet Vehicle Tracking • Sierra Wireless – MP775 • EDGE, GPRS, GSM Class 12 technology • Remote field management and firmware upgradeable • Quad-band 850, 900, 1800 & 1900 MHz operations • Operates in extreme temperatures -40C to +75C (-40F to +167F) • Meets US Military and SAE specs for vibration, shock, drop, humidity and splash • 12 Channel GPS IDEA Laboratory University of Kentucky

38. ~20 ft. blimp Up to 3000 ft. altitude Remote controlled PLZ camera system Tethered Camera Platform IDEA Laboratory University of Kentucky

39. Automatic Antenna Tracking for UAVs – Ground Station IDEA Laboratory University of Kentucky

40. Automatic Antenna Tracking for UAVs – Onboard Systems IDEA Laboratory University of Kentucky

41. Communication VHF TNC link Amateur television (ATV) downlink Objectives Surveillance Photography (surveying) Virtual reality Ground Station IDEA Laboratory University of Kentucky

42. Questions? For more information: www.engr.uky.edu/idea IDEA Laboratory University of Kentucky

43. Sources • GPS/inertial data in aerial photogrammetry - http://www.ifp.uni-stuttgart.de/forschung/photo/georef-Dateien/georef.en.html • SGI UAV demo http://www.sgi.com/products/software/performer/brew/uav.html • American Society for Photogrammetry and Remote Sensing http://www.asprs.org/publications/pers/2002journal/may/georef.html • OSSIM http://www.ossim.org/ • Real-time Lens distortion correction http://ieee.stanford.edu/ecj/docs/ECJ_demo.pdf • Matlab Camera Calibration script http://www.vision.caltech.edu/bouguetj/calib_doc/htmls/parameters.html • Kalman filtering • http://www.cs.unc.edu/~welch/kalman/ IDEA Laboratory University of Kentucky

44. IDEA Laboratory University of Kentucky