1 / 33

Registration for Augmented Reality

Registration for Augmented Reality. Neil Birkbeck 3/27/2006. Outline. What is AR? Applications Research Areas Single Plane-Based Calibration Multiple Plane-Based Calibration Other Techniques Summary. Augmented Reality (AR). Definition (according to Azuma):

giorgio
Download Presentation

Registration for Augmented Reality

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Registration for Augmented Reality Neil Birkbeck 3/27/2006

  2. Outline • What is AR? • Applications • Research Areas • Single Plane-Based Calibration • Multiple Plane-Based Calibration • Other Techniques • Summary

  3. Augmented Reality (AR) • Definition (according to Azuma): • A system which combines virtual objects with the real world in real-time. The virtual objects should be correctly registered with the real world. • Display Methods • Head Mounted Display (HMD) • Video HMD • Optical HMD • Monitor/TV

  4. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • …

  5. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • … http://www.informationinplace.com/Solutions/CaseStudies/case_RDECOM/Demos/Blast/lBlast.html

  6. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • … http://jama.ama-assn.org/cgi/content/full/292/18/2214-b/JLD40609F1

  7. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • … http://hci.rsc.rockwell.com/AutomationFair_2003/

  8. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • … http://wearables.unisa.edu.au/projects/ARQuake/www/

  9. AR Applications • Advertising • Military • Surgical/Medical • Maintenance • Entertainment • Commerce • … http://virtual.vtt.fi/multimedia/clipvf.html

  10. Research Areas • Display Technologies • improving the wearable HMD devices • Mobile Computing • Calibration/Registration • hardware • GPS, magnetic tracking • vision-based/range-based • hybrid methods • Rendering Issues • Calibration of illumination for correct/realistic shading

  11. Vision-Based Registration • Motivation • inexpensive/ubiquitous • video HMD already uses images • Accurate • pixel/sub-pixel precision possible • Images useful for recovering shading/occlusions • Potential Downsides • computational requirements • feasibility over wide-ranges

  12. Vision-Based Registration • Image features are used to register position of viewer (camera) • Existing methods can be categorized using the following characteristics: • Feature type: • natural - corners, lines, image patches. • synthetic – patterns, LEDs • 3D Position of features: • known – the 3D positions of the features are known • unknown – the 3D positions of features are unknown (e.g., similar to SFM)

  13. Vision-Based Registration Overview • When 3D positions of features are known, Xi=(xi,yi,zi), find camera parameters that align the projection of the 3D feature points with the observed 2D feature points, xi=(ui,vi). This is an optimization problem over the space of camera parameters p: argminp ∑i|f(p,Xi)-xi|2

  14. The Planar Case:A registration method for features on a plane • A specific implementation for features on a single plane using the typical perspective projection model. • Why planes? • Easy to make planar calibration patterns • Measurement of relative 2D positions of feature points on planes is straightforward. • Occur frequently on man-made structures • Rooftops, building walls, etc.

  15. Zhang’s Planar Calibration Method • Calibrate the camera using correspondences between (at least) four planar points to their 3D reference positions • The calibration method is based on determining the 2D projective Homography between the observed points and their reference position. • A 2D projective Homography is a 3x3 matrix that operates on 2D homogeneous points:

  16. Zhang’s Planar Calibration Method The 3x3 Homography defines the motion from the image coordinates of the pattern to the reference coordinates The camera calibration parameters are extracted from the recovered Homography

  17. Zhang’s Planar Calibration Method • The method is based on the following observation: • Where R is a 3x3 rotation matrix, t is a 3x1 translation vector, and K is the internal parameters of the camera.

  18. Zhang’s Planar Calibration Method… Assuming intrinsics are known:

  19. Planar Calibration Example • Augmented Reality with simple pattern: • Planar pattern detected each frame by thresholding and finding connected components • Quadrilateral shapes are warped to squares and correct orientation is found • Homography is recovered and used to calibrate camera

  20. Region-based Alternative Input Image at time t • On initialization, a user selects a plane of interest • The rectifying Homography and rectified template image are retained H Template

  21. Region-based Alternative Image at time t Image at time t+1 • When new image arrives, use image intensities to refine the Homography H H Template

  22. Region-Based SSD Tracking(Lucas-Kanade Tracking) • Mathematically, given: • a template, T, which is indexed by a set of 2D points, • Define the warp, which in the most general case is a homography: • The parameters of the warp are:

  23. Region-Based SSD Tracking • Find the parameter update that minimizes the sum of squared differences (SSD): • To minimize, first perform Taylor series Expansion:

  24. Region-Based SSD Tracking • The minimization problem is equivalent to a least-squares problem, with m equations, one for each xi: • Giving, the following:

  25. Region-Based SSD Example Image at time t+1 Image at time t Diff. Between template1 Template Warped Image at t+1

  26. Region-Based SSD Example Image derivatives w.r.t the homography parameters: Image Diff h11 h12 h13 h21 h22 h23 h31 h32 Update is Essentially a linear combination of Partial Derivative Images

  27. Region-Based SSD Example • Successive improvement after several iterations Rectified Diff. From Template SSD score 6867 2809 1799 583

  28. Region-Based AR Example • A single planar region was identified, tracked, and used to register the world coordinate frame with the camera

  29. Planar Calibration for AR • Problems with plane-based approaches: • Poor registration for objects far from the plane • Registration degrades at grazing views • Limited viewing range with single planar pattern/region • Potential Solutions: • use several planes (Buenaposada et al.) • use other feature types (Marchand et al.)

  30. Multiple Region-Based Registration • Use multiple planar regions that are registered with respect to one another (3D model) • Mathematical formulation is similar to the single plane-based SSD tracking • Update in camera parameters is influenced by all planar regions Approximate 3D model

  31. Other Approaches • Compute 3D model and register camera (Davison et al.) • Camera/User state is modeled with a position, orientation, 3D velocity, and angular velocity • Salient image features are detected and matched in subsequent frames to initialize uncertain 3D features • Extended Kalman Filter (EKF) is used to update camera state, 3D feature position, and their covariance matrices http://www.doc.ic.ac.uk/~ajd/

  32. Summary • Vision-based registration useful for AR • High accuracy is possible • Plane-based techniques simple and efficient • Use of non-planar features more stable through wider ranges

  33. References • Azuma, Ronald T. "A Survey of Augmented Reality." Presence: Teleoperators and Virtual Environments 6, 4 (August 1997), 355 - 385 • Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000. • José Miguel Buenaposada, Enrique Muñoz, Luis Baumela. Tracking heads using piecewise planar models.. Proc. of Iberian Conference on Pattern Recognition and Image Analysis, IbPRIA 2003. LNCS 2652, pp. 126-133 (ISBN 3-540-40217-9), (c) Springer-Verlag. Palma de Mallorca, Spain, June 2003. • Dana Cobzas and Peter Sturm, 3D SSD Tracking with Estimated 3D Planes,In proceedings of Computer Robot Vision (CRV05), Pages 129-134, 2005. • José Miguel Buenaposada Biencinto, Luis Baumela Molina. Real-time tracking and estimation of plane pose., In Proc. of International Conference on Pattern Recognition, ICPR 2002. Vol. II, pp. 697-700. (c) IEEE. Quebec, Canada, August 2002. • E. Marchand, F. Chaumette. Virtual Visual Servoing: a framework for real-time augmented reality. In EUROGRAPHICS 2002 Conference Proceeding, G. Drettakis, H.-P. Seidel (eds.), Computer Graphics Forum, Volume 21(3), Pages 289-298, Sarrebruck, Germany, September 2002. • Simon Baker and Iain Matthews. Lucas-Kanade 20 Years On: A Unifying Framework. International Journal of Computer Vision, Vol. 56, No. 3, March, 2004, pp. 221 - 25

More Related