Motion from image and inertial measurements
Download
1 / 114

Motion from image and inertial measurements - PowerPoint PPT Presentation


  • 116 Views
  • Uploaded on

Motion from image and inertial measurements. Dennis Strelow Carnegie Mellon University. On the web. Related materials: these slides related papers movies VRML models at: http://www.cs.cmu.edu/~dstrelow/northrop2005. Introduction (1). From an image sequence, we can recover:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Motion from image and inertial measurements' - kyle


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Motion from image and inertial measurements

Motion from image and inertial measurements

Dennis Strelow

Carnegie Mellon University


On the web
On the web

Related materials:

  • these slides

  • related papers

  • movies

  • VRML models

    at:

    http://www.cs.cmu.edu/~dstrelow/northrop2005

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20042


Introduction 1
Introduction (1)

From an image sequence, we can recover:

  • 6 degree of freedom (DOF) camera motion

  • without knowledge of the camera’s surroundings

  • without GPS

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20043


Introduction 2
Introduction (2)

Fitzgibbon

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20044


Introduction 3
Introduction (3)

  • Potential applications include:

  • modeling from video

Yuji Uchida

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20045


Introduction 4
Introduction (4)

  • micro air vehicles (MAVs)

AeroVironment Black Widow

AeroVironment Microbat

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20046


Introduction 5
Introduction (5)

  • rover navigation

Hyperion

Nister, et al.

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20047


Introduction 6
Introduction (6)

  • search and rescue robots

Rhex (movies: http://ai.eecs.umich.edu/Rhex/Movies)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20048


Introduction 7
Introduction (7)

  • NASA Personal Satellite Assistant (PSA)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 20049


Introduction 8
Introduction (8)

For these problems, we want:

  • 6 DOF motion

  • in unknown environments

  • without GPS or other absolute positioning

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200410


Introduction 81
Introduction (8)

For these problems, we want:

  • 6 DOF motion

  • in unknown environments

  • without GPS or other absolute positioning

  • using small, light, and cheap sensors

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200411


Introduction 82
Introduction (8)

For these problems, we want:

  • 6 DOF motion

  • in unknown environments

  • without GPS or other absolute positioning

  • using small, light, and cheap sensors

  • over the long term

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200412


Introduction 9
Introduction (9)

Long-term motion estimation:

  • absolute distance or time is long

  • only a small fraction of the scene is visible at any one time

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200413


Introduction 10
Introduction (10)

  • given these requirements, cameras are promising sensors…

  • …and many algorithms for motion from images already exist

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200414


Introduction 11
Introduction (11)

But, where are the systems for estimating the motion of:

over the long term?

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200415


Introduction 12
Introduction (12)

…and for automatically modeling

  • rooms

  • buildings

  • cities

    from a handheld camera?

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200416


Introduction 13
Introduction (13)

Motion from images suffers from some long-standing difficulties

This work attacks these problems by…

  • exploiting image and inertial measurements

  • robust image feature tracking

  • recognizing previously mapped locations

  • exploiting omnidirectional images

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200417


Outline
Outline

Motion from images

refresher

bundle adjustment

difficulties

Motion from image and inertial measurements

Robust image feature tracking

Long-term motion estimation

Conclusion

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200418


Motion from images refresher 1
Motion from images: refresher (1)

A two-step process is common:

  • sparse feature tracking

  • estimation

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200419


Motion from images refresher 11
Motion from images: refresher (1)

A two-step process is common:

  • sparse feature tracking

  • estimation

    Sparse feature tracking:

  • inputs: raw images

  • outputs: projections

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200420


Motion from images refresher 2
Motion from images: refresher (2)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200421


Motion from images refresher 3
Motion from images: refresher (3)

Template matching:

  • correlation tracking

  • Lucas-Kanade (Lucas and Kanade, 1981)

    Extraction and matching:

  • Harris features (Harris, 1992)

  • Scale Invariant Feature Transform (SIFT) keypoints (Lowe, 2004)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200422


Motion from images refresher 4
Motion from images: refresher (4)

The second step is estimation:

  • inputs:

    • projections

  • outputs:

    • 6 DOF camera position at the time of each image

    • 3D position of each tracked point

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200423


Motion from images refresher 5
Motion from images: refresher (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200424


Motion from images refresher 6
Motion from images: refresher (6)

  • bundle adjustment (various, 1950’s)

  • Kalman filtering (Broida, Chandrashekhar, and Chellappa, 1990)

  • variable state dimension filter (VSDF) (McLauchlan, 1996)

  • two- and three-frame methods(Hartley and Zisserman, 2000; Nister, et al. 2004)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200425


Motion from images bundle adjustment 1
Motion from images: bundle adjustment (1)

From tracking, we have the image locations xij for each point j and each image i

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200426


Motion from images bundle adjustment 2
Motion from images: bundle adjustment (2)

Suppose we also have estimates of:

  • the camera rotation ρi and translation ti at time of each image

  • 3D point positions Xj of each tracked point

    Then, we can compute reprojections:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200427


Motion from images bundle adjustment 3
Motion from images: bundle adjustment (3)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200428


Motion from images bundle adjustment 4
Motion from images: bundle adjustment (4)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200429


Motion from images bundle adjustment 5
Motion from images: bundle adjustment (5)

So, minimize:

with respect to all the ρi, ti, Xj

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200430


Motion from images bundle adjustment 51
Motion from images: bundle adjustment (5)

So, minimize:

with respect to all the ρi, ti, Xj

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200431


Motion from images difficulties 1
Motion from images: difficulties (1)

Estimation step can be very sensitive to…

  • incorrect or insufficient image feature tracking

  • camera modeling and calibration errors

  • outlier detection thresholds

  • sequences with degenerate camera motions

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200432


Motion from images difficulties 2
Motion from images: difficulties (2)

Iterative batch methods have poor convergence or may fail to converge if:

  • observations are missing

  • the initial estimate is poor

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200433


Motion from images difficulties 3
Motion from images: difficulties (3)

Recursive methods suffer from:

  • poor prior assumptions on the motion

  • poor approximations in state error modeling

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200434


Motion from images difficulties 4
Motion from images: difficulties (4)

Resulting errors are:

  • gross local errors

  • long term drift

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200435


Motion from images difficulties 5
Motion from images: difficulties (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200436


Motion from images difficulties 6
Motion from images: difficulties (6)

  • 151 images, 23 points

  • manually corrected Lucas-Kanade

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200437


Motion from images difficulties 7
Motion from images: difficulties (7)

  • squares: ground truth points

  • dash-dotted line: accurate estimate

  • solid line: image-only, bundle adjustment estimate

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200438


Outline1
Outline

Motion from images

Motion from image and inertial measurements

inertial sensors

algorithms and results

related work

Robust image feature tracking

Long-term motion estimation

Conclusion

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200439


Motion from image and inertial measurements inertial sensors 1
Motion from image and inertial measurements: inertial sensors (1)

  • inertial sensors can be integrated to estimate six degree of freedom motion

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200440


Motion from image and inertial measurements inertial sensors 2
Motion from image and inertial measurements: inertial sensors (2)

But many applications require small, light, and cheap sensors

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200441


Motion from image and inertial measurements inertial sensors 3
Motion from image and inertial measurements: inertial sensors (3)

Integrating the outputs of these low grade sensors will produce drifting motion because of:

  • noise

  • unmodeled nonlinearities

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200442


Motion from image and inertial measurements inertial sensors 4
Motion from image and inertial measurements: inertial sensors (4)

  • And, we can’t even integrate until we can separate the effects of…

  • rotation ρ

  • gravity g

  • acceleration a

  • slowly changing bias ba

  • noise n

  • …in the accelerometer measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200443


Motion from image and inertial measurements inertial sensors 5
Motion from image and inertial measurements: inertial sensors (5)

Image and inertial measurements are highly complementary

With inertial measurements we can:

  • decrease sensitivity in image-only estimates

  • establish two rotation angles without drift

  • establish the global scale

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200444


Motion from image and inertial measurements inertial sensors 51
Motion from image and inertial measurements: inertial sensors (5)

Image and inertial measurements are highly complementary

With inertial measurements we can:

  • decrease sensitivity in image-only estimates

  • establish two rotation angles without drift

  • establish the global scale

    …even with our low-grade sensors

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200445


Motion from image and inertial measurements inertial sensors 6
Motion from image and inertial measurements: inertial sensors (6)

With image measurements, we can:

  • reduce the drift in integrating inertial data

  • distinguish between…

    • rotation ρ

    • gravity g

    • acceleration a

    • bias ba

    • noise n

      …in accelerometer measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200446


Motion from image and inertial measurements algorithms and results 1
Motion from image and inertial measurements: algorithms and results (1)

This work has developed both:

  • batch

  • recursive

    algorithms for motion from image and inertial measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200447


Motion from image and inertial measurements algorithms and results 2
Motion from image and inertial measurements: algorithms and results (2)

Gyro measurements:

  • ω’, ω: measured and actual angular velocity

  • bω: gyro bias

  • n: gaussian noise

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200448


Motion from image and inertial measurements algorithms and results 3
Motion from image and inertial measurements: algorithms and results (3)

Accelerometer measurements:

  • ρ: rotation

  • a’, a: measured and actual acceleration

  • g: gravity vector

  • ba: accelerometer bias

  • n: gaussian noise

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200449


Motion from image and inertial measurements algorithms and results 4
Motion from image and inertial measurements: algorithms and results (4)

  • batch algorithm minimizes a combined error:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200450


Motion from image and inertial measurements algorithms and results 5
Motion from image and inertial measurements: algorithms and results (5)

  • image term Eimage is the same as before

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200451


Motion from image and inertial measurements algorithms and results 6
Motion from image and inertial measurements: algorithms and results (6)

  • inertial error term Einertial is:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200452


Motion from image and inertial measurements algorithms and results 61
Motion from image and inertial measurements: algorithms and results (6)

  • inertial error term Einertial is:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200453


Motion from image and inertial measurements algorithms and results 62
Motion from image and inertial measurements: algorithms and results (6)

  • inertial error term Einertial is:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200454


Motion from image and inertial measurements algorithms and results 7
Motion from image and inertial measurements: algorithms and results (7)

( : translation estimate

for image i – 1)

ti-1

( : translation estimate

for image i)

translation

ti

I(ti-1, …)

( : translation integrated

from previous estimate)

τi-1

(time of image i - 1)

τi

(time of image i)

time

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200455


Motion from image and inertial measurements algorithms and results 8
Motion from image and inertial measurements: algorithms and results (8)

translation

time

τ0

τ1

τ2

τ3

τ4

τ5

τf-3

τf-2

τf-1

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200456


Motion from image and inertial measurements algorithms and results 9
Motion from image and inertial measurements: algorithms and results (9)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200457


Motion from image and inertial measurements algorithms and results 10
Motion from image and inertial measurements: algorithms and results (10)

It(τi-1, τi ,…, ti-1)depends on:

  • τi-1, τi(known)

  • all inertial measurements for timesτi-1<τ < τi(known)

  • ρi-1, ti-1

  • g

  • bω, ba

  • camera linear velocities:vi

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200458


Motion from image and inertial measurements algorithms and results 12
Motion from image and inertial measurements: algorithms and results (12)

  • dash-dotted line: batch estimate from image and inertial

  • solid line: image-only, bundle adjustment estimate

  • squares: ground truth points

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200459


Motion from image and inertial measurements algorithms and results 13
Motion from image and inertial measurements: algorithms and results (13)

  • IEKF for the same sensors, unknowns

  • dash-dotted line: batch estimate

  • solid line: IEKF estimate

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200460


Motion from image and inertial measurements algorithms and results 14
Motion from image and inertial measurements: algorithms and results (14)

  • Difficulties with IEKF for this application:

  • prior assumptions about motion smoothness

  • cannot model relative error between adjacent camera positions

  • So, converting the batch algorithm into a variable state dimension filter (VSDF) is a promising future direction

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200461


Motion from image and inertial measurements algorithms and results 15
Motion from image and inertial measurements: algorithms and results (15)

  • IEKF assumptions on motion smoothness

  • dash-dotted line: batch estimate

  • solid line: IEKF estimate

  • left: IEKF propagation variances just right

  • right: IEKF propagation variances too strict

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200462


Motion from image and inertial measurements1
Motion from image and inertial measurements results (15)

  • Recap:

  • image, gyro, and accelerometer measurements

  • batch algorithm

  • recursive algorithm

  • experiments

    • evaluate batch and recursive algorithms

    • establish basic facts about motion from image and inertial measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200463


Outline2
Outline results (15)

Motion from images

Motion from image and inertial measurements

Robust image feature tracking

Lucas-Kanade and real sequences

The “smalls” tracker

Long-term motion estimation

Conclusion

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200464


Robust image feature tracking lucas kanade and real sequences 1
Robust image feature tracking: Lucas-Kanade and real sequences (1)

  • Combining image and inertial measurements improves our situation, but…

  • we still need accurate feature tracking tracking

  • some sequences do not come with inertial measurements

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200465


Robust image feature tracking lucas kanade and real sequences 2
Robust image feature tracking: Lucas-Kanade and real sequences (2)

  • better feature tracking for improved 6 DOF motion estimation

  • remaining results will be image-only

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200466


Robust image feature tracking lucas kanade and real sequences 3
Robust image feature tracking: Lucas-Kanade and real sequences (3)

  • Lucas-Kanade has been the go-to feature tracker for shape-from-motion

  • minimizes a correlation-like matching error

  • using general minimization

  • evaluates the matching error at only a few locations

  • subpixel resolution

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200467


Robust image feature tracking lucas kanade and real sequences 4
Robust image feature tracking: Lucas-Kanade and real sequences (4)

Additional heuristics used to apply Lucas-Kanade to shape-from-motion:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200468


Robust image feature tracking lucas kanade and real sequences 5
Robust image feature tracking: Lucas-Kanade and real sequences (5)

But Lucas-Kanade performs poorly on many real sequences…

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200469


Robust image feature tracking the smalls tracker 1
Robust image feature tracking: the “smalls” tracker (1) sequences (5)

  • smalls is a new feature tracker targeted at 6 DOF motion estimation

  • exploits the rigid scene assumption

  • eliminates the heuristics normally used with Lucas-Kanade

  • SIFT is an enabling technology here

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200470


Robust image feature tracking the smalls tracker 2
Robust image feature tracking: the “smalls” tracker (2) sequences (5)

  • First step: epipolar geometry estimation

  • use SIFT to establish matches between the two images

  • get the 6 DOF camera motion between the two images

  • get the epipolar geometry relating the two images

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200471


Robust image feature tracking the smalls tracker 3
Robust image feature tracking: the “smalls” tracker (3) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200472


Robust image feature tracking the smalls tracker 4
Robust image feature tracking: the “smalls” tracker (4) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200473


Robust image feature tracking the smalls tracker 5
Robust image feature tracking: the “smalls” tracker (5) sequences (5)

  • Second step: track along epipolar lines

  • use nearby SIFT matches to get initial position on epipolar line

  • exploits the rigid scene assumption

  • eliminates heuristic: pyramid

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200474


Robust image feature tracking the smalls tracker 6
Robust image feature tracking: the “smalls” tracker (6) sequences (5)

  • Third step: prune features

  • geometrically inconsistent features are marked as mistracked and removed

  • clumped features are pruned

  • eliminates heuristic: detecting mistracked features based on convergence, error

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200475


Robust image feature tracking the smalls tracker 7
Robust image feature tracking: the “smalls” tracker (7) sequences (5)

  • Fourth step: extract new features

  • spatial image coverage is the main criterion

  • required texture is minimal when tracking is restricted to the epipolar lines

  • eliminates heuristic: extracting only textured features

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200476


Robust image feature tracking the smalls tracker 8
Robust image feature tracking: the “smalls” tracker (8) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200477


Robust image feature tracking the smalls tracker 9
Robust image feature tracking: the “smalls” tracker (9) sequences (5)

left: odometry only

right: images only

  • average error: 1.74 m

  • maximum error: 5.14 m

  • total distance: 230 m

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200478


Robust image feature tracking the smalls tracker 10
Robust image feature tracking: the “smalls” tracker (10) sequences (5)

  • Recap:

  • exploits the rigid scene and eliminates heuristics

  • allows hands-free tracking for real sequences

  • can still be defeated by textureless areas or repetitive texture

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200479


Outline3
Outline sequences (5)

Motion from images

Motion from image and inertial measurements

Robust image feature tracking

Long-term motion estimation

proof of concept system

experiment

Conclusion

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200480


Long term motion estimation proof of concept system 1
Long-term motion estimation: proof of concept system (1) sequences (5)

  • Image-based motion estimates from any system will drift:

  • if the features we see are always changing

  • given sufficient time

  • if we don’t recognize when we’ve revisited a location

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200481


Long term motion estimation proof of concept system 2
Long-term motion estimation: proof of concept system (2) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200482


Long term motion estimation proof of concept system 3
Long-term motion estimation: proof of concept system (3) sequences (5)

  • To limit drift:

  • recognize when we’ve returned to a previous location

  • exploit the return

  • A proof of concept system demonstrates these capabilities

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200483


Long term motion estimation proof of concept system 4
Long-term motion estimation: proof of concept system (4) sequences (5)

system state S

image indices: I = {i1, …, in}

“smalls” tracker state: 2D feature history for images in I

variable state dimension filter (VSDF) state for images in I: 6 DOF camera positions, covariances for images in I3D positions for features visible in I

SIFT keypoints for image in

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200484


Long term motion estimation proof of concept system 5
Long-term motion estimation: proof of concept system (5) sequences (5)

{0}

{0, 1, 2}

{0, 1, …, 8}

0

1

2

3

4

5

6

7

8

{0, 1}

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200485


Long term motion estimation proof of concept system 6
Long-term motion estimation: proof of concept system (6) sequences (5)

{0}

{0, 1, 2}

{0, 1, …, 8}

0

1

2

3

4

5

6

7

8

{0, 1}

States:

rollback

non-rollback

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200486


Long term motion estimation proof of concept system 7
Long-term motion estimation: proof of concept system (7) sequences (5)

{0}

{0, 1, 2}

{0, 1, …, 8}

0

1

2

3

4

5

6

7

8

{0, 1}

8

States:

rollback

non-rollback

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200487


Long term motion estimation proof of concept system 8
Long-term motion estimation: proof of concept system (8) sequences (5)

{0}

{0, 1, 2}

0

1

2

3

4

5

6

7

8

{0, 1}

8

{0, 1, 2, 3, 8}

States:

rollback

non-rollback

pruned

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200488


Long term motion estimation proof of concept system 9
Long-term motion estimation: proof of concept system (9) sequences (5)

{0, …, 6, 11, 12, 17, …, 20}

17

18

19

20

11

12

13

14

0

1

2

3

4

5

6

7

8

8

9

10

11

14

15

16

17

States:

rollback

non-rollback

pruned

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200489


Long term motion estimation proof of concept system 10
Long-term motion estimation: proof of concept system (10) sequences (5)

  • When to “roll back”?

  • examine the camera covariances for the current state and the candidate rollback state

  • check the number of SIFT matches

  • extend from the candidate state

  • examine the camera covariances for the current state and the resulting extended state

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200490


Long term motion estimation experiment 1
Long-term motion estimation: experiment (1) sequences (5)

CMU FRC highbay views; 945 images total

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200491


Long term motion estimation experiment 2
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200492


Long term motion estimation experiment 21
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(first forward pass: images 0-213)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200493


Long term motion estimation experiment 22
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(first forward pass: images 0-213)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200494


Long term motion estimation experiment 23
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(first forward pass: images 0-213)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200495


Long term motion estimation experiment 24
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(first backward pass: images 214-380)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200496


Long term motion estimation experiment 25
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(second forward pass: images 381-493)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200497


Long term motion estimation experiment 26
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(second backward pass: images 494-609)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200498


Long term motion estimation experiment 27
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(third forward pass: images 610-762)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 200499


Long term motion estimation experiment 28
Long-term motion estimation: experiment (2) sequences (5)

CMU FRC highbay

(third backward pass: images 763-944)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004100


Long term motion estimation experiment 3
Long-term motion estimation: experiment (3) sequences (5)

  • normally, the system produces a general tree of states

17

18

19

20

11

12

13

14

0

1

2

3

4

5

6

7

8

8

9

10

11

14

15

16

17

States:

rollback

non-rollback

pruned

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004101


Long term motion estimation experiment 4
Long-term motion estimation: experiment (4) sequences (5)

  • for this example, the “rollback” states are restricted to the first forward pass

10

11

12

14

13

14

15

14

0

1

2

3

4

5

6

7

213

8

9

14

16

17

18

17

States:

rollback

non-rollback

pruned

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004102


Long term motion estimation experiment 5
Long-term motion estimation: experiment (5) sequences (5)

  • movie…bottom half is smalls output:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004103


Long term motion estimation experiment 6
Long-term motion estimation: experiment (6) sequences (5)

  • movie…top half is motion estimates:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004104


Long term motion estimation experiment 7
Long-term motion estimation: experiment (7) sequences (5)

  • movie…top half is motion estimates:

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004105


Outline4
Outline sequences (5)

Motion from images

Motion from image and inertial measurements

Robust image feature tracking

Long-term motion estimation

Conclusion

remaining issues

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004106


Conclusion remaining issues
Conclusion: remaining issues sequences (5)

  • all: system is experimental, not optimized for speed

  • image and inertial: VSDF

  • “smalls”: integration of gyro, more robustness to poor texture needed

  • long-term: “roll back” space, computation grow with sequence length

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004107


Thanks
Thanks! sequences (5)

Related materials:

  • these slides

  • related papers

  • movies

  • VRML models

    at:

    http://www.cs.cmu.edu/~dstrelow/northrop2005

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004108


Motion from omnidirectional images 1
Motion from omnidirectional images (1) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004109


Motion from omnidirectional images 2
Motion from omnidirectional images (2) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004110


Motion from omnidirectional images 3
Motion from omnidirectional images (3) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004111


Motion from omnidirectional images 4
Motion from omnidirectional images (4) sequences (5)

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004112


Motion from omnidirectional images 5
Motion from omnidirectional images (5) sequences (5)

left: non-rigid camera

right: rigid camera

squares: ground truth points solid: image-only estimates

dash-dotted: image-and-inertial estimates

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004113


Motion from omnidirectional images 6
Motion from omnidirectional images (6) sequences (5)

  • In this experiment:

  • omni images

  • conventional images + inertial

  • have roughly the same advantages

  • But in general:

  • inertial has some advantages that omni images alone can’t produce

  • omni images can be harder to use

Dennis Strelow -- Motion estimation from image and inertial measurements – December 13, 2004114


ad