Reconstructing Camera Positions from Visibility Information

Reconstructing Camera Positions from Visibility Information Elif Tosun Honors Thesis Presentation Advisor: Ileana Streinu May 1st, 2002

Outline • Motivation • General Problem • Some Definitions • Main Problems of Interest • Background in Rigidity • Solutions in 2D • Further Work • Summary

Motivation: MIT City Scanning Project • Goal: To reconstruct a 3D model of an urban environment using numerous images.

Motivation: MIT City Project • Method: • Robot with a mounted hemispherical camera • Images acquired at different *unknown* positions of an urban environment • Images put together to get “image feature points” using methods in Computer Vision and Computer Graphics. A corner may be an image feature point

Method (Cont’d): • Slopes between cameras and image features are inferred from use of hemispherical cameras with known local orientations. • Once the set of slopes are obtained, solution for camera and image feature positions is based on solving a linear system of equations

However… • A linear system of equations may be over or under determined. • This information is NOT known a priori • Therefore, system *sometimes* may fail to give a solution. • A SUBSET of edges that give a unique solution is needed. “… determination of the minimal set of distinct pairs needed for a non-degenerate solution is significantly more complicated and depends entirely on the topology of the adjacency graph. ” [Antone-Teller, 2000]

The Problem • Given: Visibility information as a set of slopes at n positions of a hemispherical camera • Find: Absolute coordinates of camera locations and positions of image features • Why? May be used to reconstruct a model of environment. • Problem? Over- or under-determined system of equations. Need a subset.

This thesis focuses on: • Reformulating this problem and modeling it based on its geometric/combinatorial aspects • Exploring it in 2 and 3 dimensions • Isolating several feasible sub problems and giving algorithmic solutions • Providing implementations for some

General Camera Registration Problem • Formulation using “direction networks” Data set: 3 cameras ,4 image feature points, slope known for 13 pairs Direction Network: 7 (=3+4) vertices 13 edges

Direction Network? A direction network is a graph G=(V, E) together with slope information for each edge: D =(V, E, S) with S = {se: se is the slope information for an edge e E}

General Camera Registration Problem Given a direction network D and the location of a subset of vertices V’ of V, find the location of all vertices v V of G. Notes: 1. We don’t treat image features and camera positions differently. Both sets together form the set of vertices V of G. 2. Location of a subset of vertices is needed to fix a frame of reference.

Definitions • Realization of a direction network: A geometric embedding of G of D. • i.e. set of points P s.t. if (ij)E then the slope between points piandpj issij

Definitions • Realization of a direction network: A geometric embedding of G of D. • i.e. set of points P s.t. if (ij)E then the slope between points piandpj is sij y-axis x-axis (0,0)

Definitions • Mixed “direction and length” network: A direction network of the following form: MD = (V, v1, E1, E2, S, L) where: E1 = set of edges for which slopes are given E2= set of edges for which lengths are given S = {se: se is the slope for edge eE1} L = {le: le is the length for edge eE2} v1 = a unique vertex that is pinned down (at origin)

How to solve for an MD system To solve for a mixed “direction and length” network… A) Associate a linearequation to each slope information… i.e. for each slope sij between points pi=(xi, yi) and pj=(xj, yj): (yj-yi) – sij(xj-xi) = 0

B) And associate a quadratic equation to each length information i.e. for each length lij between points pi=(xi, yi) and pj=(xj, yj): (xj-xi)2 + (yi-yj)2 = lij2

A particular case… The particular case of an MD that we will be considering is the following: There exists an edge e, s.t. • both the length (le) and the slope information (se) is known • one end point of e is pinned down • e is the only edge in E2 i.e. it’s as if two vertices are pinned down: se So, we’re only dealing with a linear system of equations le

More Definitions • Tight direction network: A direction network where there is a unique solution to the system.

Loose direction network: A direction network where the system is under-determined.

Loose direction network: A direction network where the system is under-determined. It has a parallel redrawing!

Over-determined direction network: A direction network where there are more edges than needed to have a unique solution:

Non-realizable direction network: A direction network with no solution where the system is over-determined and the constraints are incompatible

Main Problems • Problem 1: Given a direction network, decide whether it’s tight or not. • Problem 2: Given a loose direction network, find a set of edges, that needs to be added, so that it becomes tight. - if possible

Problem 3: Given an over-determined direction network: • Find a set of edges that need to be removed so that it becomes tight – if possible • Find the set of edges to be removed to get ‘the best’ tight framework. • Get an approximation of ‘the best’ tight framework.

Why “if possible” for Problems 2 and 3: • The data set might not have the edges necessary for tightness • The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined

Why “if possible” for Problems 2 and 3: • The data set might not have the edges necessary for tightness • The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined Under-determined sub-graph No edge removal can make it tight

Why “if possible” for Problems 2 and 3: • The data set might not have the edges necessary for tightness • The given direction network might be partially over- or under-determined All induced sub-graphs are under-determined Over-determined sub-graph No edge addition can make it tight

Background in RigidityGeometric Rigidity • 2 dual categories: • Fixed Edge Lengths • Fixed Directions (“parallel redrawing”) • Duality in 2D: Properties of one hold for the other as well!

Fixed Length Rigidity:Definitions • A framework is a graph G=(V, E) together with length information for each edge: F =(V, E, L) with L = {le: le is the length information for an edge e E} le

Fixed Length Rigidity:Definitions • A minimally rigid framework • A flexible framework (infinitesimal rigidity – velocities)

Fixed Length Rigidity:Definitions • An over-braced framework • An unrealizable framework

Fixed Length vs. Fixed Direction • Direction Network • Tight • Loose • Framework • Minimally Rigid • Flexible

Fixed Length vs. Fixed Direction • Over-determined • Unrealizable • Over-braced • Unrealizable

Fixed Length vs. Fixed Direction • Find point coordinates • Parallel Redrawing Matrix (M) (2nxm). • Info on slopes • Find a non-trivial set of velocities • Rigidity Matrix (M) (2nxm). • Info on derived from point coordinates

Solution: Mv = b Why? (fixed rods, perpendicular velocity vectors) (pi-pj).(vi-vj) = 0 Solution: Mv = b Why? (line equations) (xi-xj)mij –(yi-yj) = 0 Fixed Length vs. Fixed Direction V = column vector of unknowns, b = column vector of 0s. pj mij pi

Solution: Mv = b Why? (fixed rods, perp. velocity) (pi-pj).(vi-vj) = 0 Solution: Mv = b Why? (line equations) (xi-xj)mij –(yi-yj) = 0 Fixed Length vs. Fixed Direction V = column vector of unknowns, b = column vector of 0s. pj mij pi

Rigidity in the Plane:Combinatorial Rigidity • Based on underlying graph, no numeric data • Laman’s Theorem: Given a minimally rigid graph G=(V, E) with n vertices, m edges: • m = 2n-3 • For every sub graph G’ of k vertices has <= 2k-3 edges

Generic Rigidity & Tightness • A generic minimally rigid framework(or tight dir. network) : minimally rigid (or tight) for generic positions of points. Generic Non-generic

Given a direction network, decide if tight or not Solution to Problem 1(Decision Problem) • Geometric Approach : Using parallel redrawing matrix If there exists a solution to Mv=B Check dimension, if =0, TIGHT if > 0, UNDER-DETERMINED Else UNREALIZABLE

Decision ProblemGeometric Approach (Cont’d) • PROBLEM: In case of NOISE, geometric approach may return: Unrealizable (although if erroneous edges are not used, there might be a solution) • So : we need a subset of these edges necessary and sufficient for a unique solution • Use Combinatorial Approach to get a subset, then use Geometric Approach

Solution to Problem 1(Decision Problem) • Combinatorial Approach Main Idea: • Tight direction network  has a unique solution *most* of the time • Underlying graph of a tight D.N. is a minimally rigid graph • Find a way of checking if a given graph is minimally rigid or not

Decision Problem:Combinatorial Approach • Sugihara’s Algorithm, based on bipartite matchings • Associate a bipartite graph to original • Modify it for each edge and check for complete bipartite matching • Mathematica Notebook

Complexity O(n1.5).O(n2) =O(n 3.5) Decision Problem:Combinatorial Approach: Sugihara’s Alg.

Solution to Problem 2(Extension) • Problem: Given an under-determined direction network, find a set of edges to extend it to a tight graph (if possible).

Extension ProblemThe Algorithm • Natural Approach: A Greedy Algorithm • Add one edge at a time, use “decision” algorithm to test • Proof of Correctness!

Matroid Theory and Rigidity Definition: A matroidis an ordered pair M=(S, l) satisfying the following conditions • S is a finite non empty set • l is a non empty family of subsets of S called the independent subsets, s.t. if B l and AB, then A l • If Al, and |A|<|B|, then there exists an elt x  (B-a) s.t. A  {x} l

Matroid Theory and Rigidity 2 Important Properties: 1. Result by Sugihara: “Frameworks demonstrate matroidal properties.” 2. “For matroidal structures, greedy algorithms return optimal solutions correctly” Because of these two properties, our Greedy Approach returns a correct solution

Reconstructing Camera Positions from Visibility Information