Wide Angle Camera Calibration and Real-time Distortion Correction

1. Wide Angle Camera Calibration and Real-time Distortion Correction Vitaliy Orekhov Imaging, Robotics, & Intelligent Systems Laboratory The University of Tennessee April 28, 2006

2. Outline • Task 1 • Math472 – Modern Transforms • Task 2 • ECE571 – Pattern Classifications • Task 3 • ECE574 –Test, Evaluate, and Transfer Camera Calibration Code to C++ • Real time distortion correction • GUI to perform real time distortion correction on an image sequence • Task 4 • ECE671 –Survey on Wide Angle Camera Calibration and Image Correction • Survey paper outline • Compare a recent journal article to C. Broaddus thesis

3. NIKON 14MM f/2.8D ED Ultra Wide Angle AF Nikkor Lens www.amazon.com Wide Angle Camera Calibration • It is a necessary step in 3D computer vision to extract metric information from 2D images. • Before the camera can be used for precise computer vision applications, the camera needs to be characterized. • How does a point in 3D world coordinates get projected onto the camera imaging plane? • Easier to map local information for visual search, navigation, or detection.

4. Survey on Wide Angle Camera Calibration and Image Correction • Challenging to go into details and compare the results. • Lack of standardized notation • Various approaches to evaluate accuracy • Distortion models • Many ways to classify wide angle camera calibration. • [Salvi02] J. Salvi, X. Armangue, and J. Batlle, “A comparative review of camera calibrating methods with accuracy evaluation,” Patter Recogntion, pp. 1617-1635, July 2002. • Using geometric objects / Test range calibration • Non-metric calibration / Using geometric properties • Point correspondence • Straight lines • Object of revolution • Self-calibration • Linear calibration / Closed form solution / Non-iterative • Non-linear calibration / Iterative • Two step methods • Offline calibration • Using 2D and 3D object • Online calibration • Plumb line method • Rigidity assumption • 3D reference objects • 2D reference objects • 1D line based calibration • Self-Calibration • Manual methods • Semi-Automatic methods • Automatic methods

5. Survey Outline I. Introduction a. Motivation b. Background c. Problem definition • Paper outline II. Wide Angle Camera Models/ Omnidirectional Systems a. Modeling lenses b. Modeling distortion i. Distortion models ii. Finding distortion center III. Classification Methods IV. Calibration Methods with Distortion Consideration a. Test range calibration b. Non-metric calibration • Self-calibration V. Radial Distortion Calibration a. Estimating the center of distortion b. Using lines c. Point correspondence VII. Concluding Remarks

6. Radial distortion Radial position Graph generated from 19 images with resolution of 640x480. [Hartley05] [Hartley05] R. I. Hartley and S. B. Kang, “Parameter-free Radial Distortion Correction with Centre of Distortion Estimation,” Computer Vision, 2005. ICCV 2005. Tenth IEEE International Conference on , vol.2, no.pp. 1834- 1841, 17-20 Oct. 2005 • Method to simultaneously calibrate the radial distortion function of a camera and other internal calibration parameters. • Method relies on the use of a planar calibration pattern. • Radial distortion is found in a parameter-free way, not relying on any particular distortion model. • Method relies on accurate estimation of center of distortion. [Broaddus05] C. Broaddus, “Universal Geometrci Camera Calibration with Statistical Model Selection,” Masters Thesis, Department of Electrical Engineering, The Unverstity of Tennessee, Knoxville, TN 2005.

7. distorted image point • undistorted image point • - known coordinates on planar calibration pattern Distortion Center • Center of distortion is found by computing the fundamental matrix relating the known coordinates of points on a planer calibration pattern and the extracted points from the image. • Radial distortion forces ideal points to be expanded towards or away from the center of distortion. Epipole in one view is the camera center of the other view. Center of radial expansion is the epipole.

8. Image Homographies • Estimate homography which maps calibration pattern to the captured image. • Coordinates of image are transformed so that the center of radial distortion is at the origin. • 2 assumptions: • Distortion is radially symmetric • Radial distance of distorted points from distortion center is a monotonic function of the distance of undistorted points.

9. Radial distortion Radial position Graph generated from 19 images with resolution of 640x480. [Hartley05] Image Homographies • Monotonic smooth curve. • Linear Least-Squares Problem for finding the total squared variation of function. • With some simplifications image homographies can be uniquely estimated by a linear algorithm. • Can also use assumption of local linearity.

10. Radial distortion Radial position Graph generated from 19 images with resolution of 640x480. [Hartley05] Distortion Estimation • Curve can be used to correct distortion in any other image captured with the same camera. • Approximate the curve by a distortion function.

11. Complete Calibration • With homographies between the calibration pattern and images known, true undistorted projection of each calibration point can be made onto the image. • Given the homographies for at least three images, Zhang’s calibration method for a projective camera is used to find internal camera parameters. • Non-iterative rapid method. Could be used to initialize a bundle-adjustment algorithm or as a stand alone algorithm for camera calibration not requiring extreme precision. • Could be improved with iterative refinement at different steps of the process.

12. Comparison

13. www.oksolar.com www.fujinon.de Correcting Distortion • Correcting Distortion on an image sequence in real-time • IQeye3 Network Camera • 1.8M pixels/sec JPEG • 1/2" 1288 x 968 progressive scan CMOS digital imager • 50 fps (frames per second) at 160 by 120 • 19 fps at 320 by 240 • 5.4 fps at 640 by 480 • 3.6 fps at 800 by 600 • 1.3 fps at 1288 by 968 • YV2.2 x 1.4A-SA2 Lens • Fish-eye vari-focal lens with the horizontal field angle of 185-94 degrees (when used on 1/3 cameras)

14. Image Sequence Correction Diagram

15. Camera Calibration • Capturing sequence of images with calibration pattern. • Extracting coordinates using Harris corner detections. • Use MATLAB calibration code written by Chris Broaddus. • Extract camera matrix • Distortion parameters • Radial and tangential distortion • Single polynomial approximation used to model the lens projections.

16. Image size: 620x620 185 deg. fov Image size: 620x620 80 deg. fov Image Correction • Code written in C++ • Creates a look up table based on the camera matrix and distortion coefficients. • Code works with most commonly used distortion models: • Look up table is used to correct each image captured from the camera in real-time.

17. GUI

18. Wide Angle Distortion Correction • Image size 312x312 • Frame rate from camera: 14 fps. • Frame rate with distortion correction: 10 fps. • Smallest angle-of-view setting (about 80 deg.)

19. Fish-eye Distortion Correction • Image size 312x312 • Frame rate from camera: 14 fps. • Frame rate with distortion correction: 10 fps. • Widest angle-of-view setting (185deg.)

20. Wide Angle Distortion Correction • Original Image size 312x312 • Restored Image size 428x428 • Frame rate: 14 fps. • Smallest angle-of-view setting (about 80 deg.)

21. Processing Time Comparison 1* D. Eadie, F. Shevlin, A, Nisbet “Correction of Geometric Image Distortion Using FPGA’s” SPIE Proceeding Opto-Ireland, Vol. 4877,pp. 28-37, March 2003.

22. Summary

23. Thank you Suggestions/Comments/Questions