A Flexible New Technique for Camera Calibration Zhengyou Zhang

1. A Flexible New Technique for Camera CalibrationZhengyou Zhang Sung Huh CSPS 643 Individual Presentation 1 February 25, 2009

2. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

3. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

4. Introduction • Extract metric information from 2D images • Much work has been done by photogrammetry and computer vision community • Photogrammetric calibration • Self-calibration

5. Photogrammetric Calibration(Three-dimensional reference object-based calibration) • Observing a calibration object with known geometry in 3D space • Can be done very efficiently • Calibration object usually consists of two or three planes orthogonal to each other • A plane undergoing a precisely known translation is also used • Expensive calibration apparatus and elaborate setup required

6. Self-Calibration • Do not use any calibration object • Moving camera in static scene • The rigidity of the scene provides constraints on camera’s internal parameters • Correspondences b/w images are sufficient to recover both internal and external parameters • Allow to reconstruct 3D structure up to a similarity • Very flexible, but not mature • Cannot always obtain reliable results due to many parameters to estimate

7. Other Techniques • Vanishing points for orthogonal directions • Calibration from pure rotation

8. New Technique from Author • Focused on a desktop vision system (DVS) • Considered flexibility, robustness, and low cost • Only require the camera to observe a planar pattern shown at a few (minimum 2) different orientations • Pattern can be printed and attached on planer surface • Either camera or planar pattern can be moved by hand • More flexible and robust than traditional techniques • Easy setup • Anyone can make calibration pattern

9. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

10. Notation • 2D point, • 3D point, • Augmented Vector, • Relationship b/w 3D point M and image projection m (1)

11. Notation • s: extrinsic parameters that relates the world coord. system to the camera coord. System • A: Camera intrinsic matrix • (u0,v0): coordinates of the principal point • α,β: scale factors in image u and v axes • γ: parameter describing the skew of the two image

12. Homography b/w the Model Plane and Its Image • Assume the model plane is on Z = 0 • Denote ith column of the rotation matrix R by ri • Relation b/w model point Mand image m • His homography and defined up to a scale factor (2)

13. Constraints on Intrinsic Parameters • Let H be H = [h1 h2 h3] • Homography has 8 degrees of freedom & 6 extrinsic parameters • Two basic constraints on intrinsic parameter (3) (4)

14. Geometric Interpretation • Model plane described in camera coordinate system • Model plane intersects the plane at infinity at a line

15. Geometric Interpretation • x∞is circular point and satisfy , or a2 + b2 = 0 • Two intersection points • This point is invariant to Euclidean transformation

16. Geometric Interpretation • Projection of x∞ in the image plane • Point is on the image of the absolute conic, described by A-TA-1 • Setting zero on both real and imaginary parts yield two intrinsic parameter constraints

17. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

18. Calibration • Analytical solution • Nonlinear optimization technique based on the maximum-likelihood criterion

19. Closed-Form Solution • Define B = A-TA-1≡ • B is defined by 6D vector b (5) (6)

20. Closed-Form Solution • ith column of H = hi • Following relation hold (7)

21. Closed-Form Solution • Two fundamental constraints, from homography, become • If observed n images of model plane • V is 2n x 6 matrix • Solution of Vb = 0 is the eigenvector of VTV associated w/ smallest eigenvalue • Therefore, we can estimate b (8) (9)

22. Closed-Form Solution • If n ≥ 3, unique solution b defined up to a scale factor • If n = 2, impose skewless constraint γ = 0 • If n = 1, can only solve two camera intrinsic parameters, αandβ, assumingu0andv0are known and γ = 0

23. Closed-Form Solution • Estimate B up to scale factor, B = λATA-1 • B is symmetric matrix defined by b • B in terms of intrinsic parameter is known • Intrinsic parameters are then

24. Closed-Form Solution • Calculating extrinsic parameter from Homography H = [h1 h2 h3] = λA[r1 r2 t] • R = [r1r2r3] does not, in general, satisfy properties of a rotation matrix because of noise in data • R can be obtained through singular value decomposition

25. Maximum-Likelihood Estimation • Given n images of model plane with m points on model plane • Assumption • Corrupted Image points by independent and identically distributed noise • Minimizing following function yield maximum likelihood estimate (10)

26. Maximum-Likelihood Estimation • is the projection of point Mj in image i • Ris parameterized by a vector of three parameters • Parallel to the rotation axis and magnitude is equal to the rotation angle • Rand r are related by the Rodrigues formula • Nonlinear minimization problem solved with Levenberg-Marquardt Algorithm • Require initial guess

27. Calibration Procedure • Print a pattern and attach to a planar surface • Take few images of the model plane under different orientations • Detect feature points in the images • Estimate five intrinsic parameters and all the extrinsic parameters using the closed-form solution • Refine all parameters by obtaining maximum-likelihood estimate

28. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

29. Experimental Results • Off-the-shelf PULNiX CCD camera w/ 6mm lense • 640 x 480 image resolution • 5 images at close range (set A) • 5 images at larger distance (set B) • Applied calibration algorithm on set A, set B and Set A+B

30. Experimental Result Angle b/w image axes

31. Experimental Resulthttp://research.microsoft.com/en-us/um/people/zhang/calib/

32. Outline • Introduction • Equations and Constraints • Calibration and Procedure • Experimental Results • Conclusion

33. Conclusion • Technique only requires the camera to observe a planar pattern from different orientation • Pattern could be anything, as long as the metric on the plane is known • Good test result obtained from both computer simulation and real data • Proposed technique gains considerable flexibility

34. AppendixEstimating Homography b/w the Model Plane and its Image • Method based on a maximum-likelihood criterion (Other option available) • Let Mi and mi be the model and image point, respectively • Assume mi is corrupted by Gaussian noise with mean 0 and covariance matrix Λmi

35. Appendix • Minimizing following function yield maximum-likelihood estimation of H • where with = ith row of H

36. Appendix • Assume for all i • Problem become nonlinear least-squares one, i.e. • Nonlinear minimization is conducted with Levenberg-Marquardt Algorithm that requires an initial guess with following procedure to obtain

37. Appendix • Let Then (2) become • n above equation with given n point and can be written in matrix equation as Lx = 0 • L is 2n x 9 matrix • x is define dup to a scale factor • Solution of xLTL associated with the smallest eigenvalue

38. Appendix • Elements of L • Constant 1 • Pixels • World coordinates • Multiplication of both

39. Possible Future Work • Improving distortion parameter caused by lens distortion

40. Question?