Off the shelf vision based mobile robot sensing
Download
1 / 61

Off-the-Shelf Vision-Based Mobile Robot Sensing - PowerPoint PPT Presentation


  • 168 Views
  • Uploaded on

Off-the-Shelf Vision-Based Mobile Robot Sensing. Zhichao Chen Advisor: Dr. Stan Birchfield Clemson University. Vision in Robotics. A robot has to perceive its surroundings in order to interact with it. Vision is promising for several reasons: Non-contact (passive) measurement Low cost

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Off-the-Shelf Vision-Based Mobile Robot Sensing' - makan


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Off the shelf vision based mobile robot sensing l.jpg

Off-the-Shelf Vision-Based Mobile Robot Sensing

Zhichao Chen

Advisor: Dr. Stan Birchfield

Clemson University


Vision in robotics l.jpg
Vision in Robotics

  • A robot has to perceive its surroundings in order to interact with it.

  • Vision is promising for several reasons:

    • Non-contact (passive) measurement

    • Low cost

    • Low power

    • Rich capturing ability


Project objectives l.jpg
Project Objectives

Path following: Traverse a desired trajectory in both indoor and outdoor environments.

1. “Qualitative vision-based mobile robot navigation”, Proceedings of the IEEE International Conference on Robotics and Automation (ICRA),2006.

2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 25(3):749-754, June 2009.

Person following:Follow a person in a cluttered indoor environment.

“Person Following with a Mobile Robot Using Binocular Feature-Based

tracking”, Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),2007

Door detection:Build a semantic map of the locations of doors as the robot drives down a corridor.

“Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR),2008,


Motivation for path following l.jpg
Motivation for Path Following

  • Goal: Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments

  • Applications: courier, delivery, tour guide, scout robots

  • Previous approaches:

    • Image Jacobian [Burschka and Hager 2001]

    • Homography [Sagues and Guerrero 2005]

    • Homography (flat ground plane) [Liang and Pears 2002]

    • Man-made environment [Guerrero and Sagues 2001]

    • Calibrated camera [Atiya and Hager 1993]

    • Stereo cameras [Shimizu and Sato 2000]

    • Omni-directional cameras [Adorni et al. 2003]


Our approach to path following l.jpg
Our Approach to Path Following

  • Key intuition: Vastly overdetermined system(Dozens of feature points, one control decision)

  • Key result: Simple control algorithm

    • Teach / replay approach using sparse feature points

    • Single, off-the-shelf camera

    • No calibration for camera or lens

    • Easy to implement (no homographies or Jacobians)


Preview of results l.jpg
Preview of Results

milestone image

current image

top-down view

overview


Tracking feature points l.jpg
Tracking Feature Points

Kanade-Lucas-Tomasi (KLT) feature tracker

  • Automatically selects features using eigenvalues of 2x2 gradient covariance matrix

  • Automatically tracks features by minimizing sum of squared differences (SSD) between consecutive image frames

  • Augmented with gain and bias to handle lighting changes

  • Open-source implementation

gradient of image

unknown displacement

gray-level images

[http://www.ces.clemson.edu/~stb/klt]


Teach replay l.jpg

goal feature

initial feature

goal feature

current feature

Teach-Replay

track

features

detect

features

destination

Teaching Phase

start

compare

features

track

features

Replay Phase


Qualitative decision rule l.jpg
Qualitative Decision Rule

Landmark

feature

image plane

Robot at goal

uGoal

uCurrent

funnel lane

No evidence“Go straight”

Feature is to the right |uCurrent| > |uGoal| “Turn right”

Feature has changed sides sign(uCurrent) ≠ sign(uGoal)  “Turn left”


The funnel lane at an angle l.jpg

α

α

α

The Funnel Lane at an Angle

Landmark

feature

image plane

Robot at goal

funnel lane

No evidence“Go straight”

Feature is to the right “Turn right”

Side change “Turn left”


A simplified example l.jpg
A Simplified Example

Landmark

feature

Robot at goal

funnel lane

funnel lane

funnel lane

funnel lane

“Go straight”

“Go straight”

“Go straight”

“Turn right”

“Turn left”

“Go straight”


The funnel lane created by multiple feature points l.jpg
The funnel Lane Created by Multiple Feature Points

Landmark #2

Landmark #1

Landmark #3

α

α

Feature is to the right “Turn right”

No evidence“Do not turn”

Side change “Turn left”


Qualitative control algorithm l.jpg
Qualitative Control Algorithm

Funnel constraints:

uGoal

Desired heading

where φ is the signed distance between the uC and uD


Incorporating odometry l.jpg
Incorporating Odometry

Desired heading

Desired heading

from ith feature point

Desired heading

from odometry

N is the number of the features;


Overcoming practical difficulties l.jpg
Overcoming Practical Difficulties

To deal with rough terrain:

Prior to comparison, feature coordinates are warped to

compensate for a non-zero roll angle about the optical axis by

applying the RANSAC algorithm.

To avoid obstacles:

The robot detects and avoids an obstacle by sonar, and the

odometry enables the robot to roughly return to the path.

Then the robot converges to the path using both odometry

and vision.


Experimental results l.jpg
Experimental Results

milestone image

current image

top-down view

overview

Videos available at http://www.ces.clemson.edu/~stb/research/mobile_robot


Experimental results17 l.jpg
Experimental Results

milestone image

current image

top-down view

overview

Videos available at http://www.ces.clemson.edu/~stb/research/mobile_robot



Experimental results avoiding an obstacle l.jpg
Experimental Results:Avoiding an Obstacle


Experimental results20 l.jpg
Experimental Results

Indoor

Outdoor

Imaging Source Firewire camera

Logitech Pro 4000 webcam


Project objectives21 l.jpg
Project Objectives

Path following:Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments.

1. “Qualitative vision-based mobile robot navigation”, Proceedings of the

IEEE International Conference on Robotics and Automation (ICRA),2006.

2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 2009

Person following:Enable a mobile robot to follow a person in a cluttered indoor environment by vision.

“Person Following with a Mobile Robot Using Binocular Feature-Based

tracking”, Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),2007

Door detection:Detect doors as the robot drives down a corridor.

“Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR),2008


Motivation l.jpg
Motivation

  • Goal:Enable a mobile robot to follow a person in a cluttered indoor environment by vision.

  • Previous approaches:

    • Appearance properties: color, edges.[Sidenbladh et al. 1999, Tarokh and Ferrari 2003, Kwon et al. 2005]

      • Person has different color from background or faces camera.

      • Lighting changes.

    • Optical flow.[Piaggio et al 1998, Chivilò et al. 2004]

      • Drift as the person moves with out-of-plane rotation

    • Dense stereo and odometry. [Beymer and Konolige 2001]

      • difficult to predict the movement of the robot (uneven surfaces, slippage in the wheels).


Our approach l.jpg
Our approach

  • Algorithm:Sparse stereo based on Lucas-Kanade feature tracking.

  • Handles:

    • Dynamic backgrounds.

    • Out-of-plane rotation.

    • Similar disparity between the person and background.

    • Similar color between the person and background.



Detect 3d features of the scene cont l.jpg

The size of each square indicates the

horizontal disparity of the feature.

Detect 3D features of the scene( Cont. )

  • Features are selected in the left image IL and matched in the right image IR.

Left image

Right image



Detecting faces l.jpg
Detecting Faces

  • The Viola-Jones frontal face detector is applied.

  • This detector is used both to initialize the system and to enhance robustness when the person is facing the camera.

Note: The face detector is not necessary in our system.


Overview of removing background l.jpg
Overview of Removing Background

1) using the known disparity of the person in the previous image frame.

2) using the estimated motion of the background.

3) using the estimated motion of the person


Remove background step 1 using the known disparity l.jpg

Background features

Foreground features in step 1

Remove BackgroundStep 1: Using the known disparity

Discard features for which

where is the known disparity of the person in the previous frame,

and is the disparity of a feature at time t .

Original features


Remove background step 2 using background motion l.jpg

Foreground features after step 2

Foreground features with similar

disparity in step 1

Remove BackgroundStep 2: Using background motion

  • Estimate the motion of the background by computing a 4 × 4 affine transformation matrix H between two image frames at times t and t + 1:

  • Random sample consensus (RANSAC) algorithm is

  • used to yield dominant motion.


Remove background step 3 using person motion l.jpg
Remove BackgroundStep 3: Using person motion

  • Similar to step 2, the motion model of the person is calculated.

    • The size of the person group should be the biggest.

    • The centroid of the person group should be proximate to the previous location of the person.

Foreground features after step 2

Foreground features after step 3






Project objectives36 l.jpg
Project Objectives

Path following:Enable mobile robot to follow a desired trajectory in both indoor and outdoor environments.

1. “Qualitative vision-based mobile robot navigation”, Proceedings of the

IEEE International Conference on Robotics and Automation (ICRA),2006.

2. “Qualitative vision-based path following”, IEEE Transactions on Robotics, 2009

Person following:Enable a mobile robot to follow a person in a cluttered indoor environment by vision.

“Person Following with a Mobile Robot Using Binocular Feature-Based

tracking”, Proceedings of the IEEE International Conference on

Intelligent Robots and Systems (IROS),2007

Door detection:Detect doors as the robot drives down a corridor.

“Visual detection of lintel-occluded doors from a single camera”, IEEE Computer Society Workshop on Visual Localization for Mobile Platforms (in association with CVPR),2008


Motivation for door detection l.jpg
Motivation for Door Detection

Metric map

Topological map

Either way, doors are semantically meaningfullandmarks


Previous approaches to detecting doors l.jpg
Previous Approaches to Detecting Doors

  • Range-based approaches

    • sonar [Stoeter et al.1995], stereo [Kim et al. 1994], laser [Anguelov et al. 2004]

  • Vision-based approaches

fuzzy logic

[Munoz-Salinas et al. 2004]

neural network

[Cicirelli et al 2003]

color segmentation

[Rous et al. 2005]

  • Limitations:

  • require different colors for doors and walls

  • simplified environment (untextured floor, no reflections)

  • limited viewing angle

  • high computational load

  • assume lintel (top part) visible


What is lintel occluded l.jpg
What is Lintel-Occluded?

Lintel-occluded

  • post-and-lintel architecture

  • camera is low to ground

  • cannot point upward b/c obstacles

lintel

post


Our approach40 l.jpg
Our Approach

Assumptions:

• Both door posts are visible

• Posts appear nearly vertical

• The door is at least a certain width

Key idea: Multiple cues are necessary for robustness (pose, lighting, …)



Pairs of vertical lines l.jpg
Pairs of Vertical Lines

vertical lines

detected lines

Canny edges

non-vertical lines

  • Edges detected by Canny

  • Line segmentsdetected by modified Douglas-Peucker algorithm

  • Clean up (merge lines across small gaps, discard short lines)

  • Separate vertical and non-vertical lines

  • Door candidates given by all the vertical line pairs whose spacing is within a given range


Homography l.jpg
Homography

In the image

In the world

(x, y)

(x’, y’)


Prior model features width and height l.jpg

?

?

(0,y)

(x, y)

(0,0)

(0,0)

(x,0)

Prior Model Features: Width and Height

Principal point


An example l.jpg
An Example

As the door turns, the bottom corner traces an ellipse

(projective transformation of circle is ellipse)

But not horizontal


Data model posterior features l.jpg
Data Model (Posterior) Features

Placement of top and

bottom edges (g2 , g3)

Image gradient along edges (g1)

Color (g4)

Vanishing point (g7)

Kick plate (g6)

texture (g5)

and two more…


Data model features cont l.jpg
Data Model Features (cont.)

Intensity along the line

darker (light off)

positive

brighter (light on)

negative

no gap

Bottom gap(g8)


Data model features cont48 l.jpg
Data Model Features (cont.)

Slim “U”

vertical door lines

wall

door

wall

Lleft

bottom door edge

intersection line

of wall and floor

extension of

intersection line

LRight

ε

floor

Concavity(g9)


Two methods to detect doors l.jpg
Two Methods to Detect Doors

Training images

Adaboost

Weights of

features

Weights of

features

The strong classifier

Bayesian formulation

(yields better results)


Bayesian formulation l.jpg

image

image

door

door

Bayesian Formulation

Taking the log likelihood,

Data model

Prior model


Mcmc and ddmcmc l.jpg
MCMC and DDMCMC

  • Markov Chain Monte Carlo (MCMC) is used here to maximize probability to detect door (like random walk through state space of doors)

  • Data driven MCMC (DDMCMC) is used to speed up computation

    • doors appear more frequently at the position close to the vertical lines

    • the top of the door is often occluded or a horizontal line closest to the top

    • the bottom of the door is often close to the wall/floor boundary.






Results l.jpg
Results

  • 25 different buildings

  • 600 images:

  • 100 training

  • 500 testing

  • 91.1% accuracy with

  • 0.09 FP per image

Speed: 5 fps 1.6GHz (unoptimized)


False negatives and positives l.jpg
False Negatives and Positives

strong reflection

concavity and bottom gap tests fail

distracting reflection

two vertical lines

unavailable

concavity erroneously detected

distracting reflection


Navigation in a corridor l.jpg
Navigation in a Corridor

  • Doors were detected and tracked from frame to frame.

  • Fasle positives are discarded if doors were not repeatedly detected.



Conclusion l.jpg
Conclusion

  • Path following

    • Teach-replay, comparing image coordinates of feature points (no calibration)

    • Qualitative decision rule (no Jacobians, homographies)

  • Person following

    • Detects and matches feature points between a stereo pair of images and between successive images.

    • RANSAC-based procedure to estimate the motion of each region

    • Does not require the person to wear a different color from the background.

  • Door detection

    • Integrate a variety of features of door

    • Adaboost training and DDMCMC.


Future work l.jpg
Future Work

  • Path following

    • Incorporating higher-level scene knowledge to enable obstacle avoidance and terrain characterization

    • Connecting multiple teaching paths in a graph-based framework to enable autonomous navigation between arbitrary points.

  • Person following

    • Fusing the information with additional appearance-based information ( template or edges) .

    • Integration with EM tracking algorithm.

  • Door detection

    • Calibrate the camera to enable pose and distance measurements to facilitate the building of a geometric map.

    • Integrated into a complete navigation system that is able to drive down a corridor and turn into a specified room