mobile motion tracking using onboard camera l.
Skip this Video
Loading SlideShow in 5 Seconds..
Mobile Motion Tracking using Onboard Camera PowerPoint Presentation
Download Presentation
Mobile Motion Tracking using Onboard Camera

Loading in 2 Seconds...

play fullscreen
1 / 53

Mobile Motion Tracking using Onboard Camera - PowerPoint PPT Presentation

  • Uploaded on

Mobile Motion Tracking using Onboard Camera Supervisor Prof. LYU, Rung Tsong Michael Presented by Lam Man Kit, Wong Yuk Man Outline 1. Motivation & Objective 2. Motion Input 3. Previous Work 4. Improvement & New Components 5. Applications 6. Experimental Result 7. Demo Motivation

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Mobile Motion Tracking using Onboard Camera' - niveditha

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
mobile motion tracking using onboard camera

Mobile Motion Tracking using Onboard Camera


Prof. LYU, Rung Tsong Michael

Presented by

Lam Man Kit, Wong Yuk Man


1. Motivation & Objective

2. Motion Input

3. Previous Work

4. Improvement & New Components

5. Applications

6. Experimental Result

7. Demo

  • Rapid increase in the use of camera-phone.
  • Camera-phones’ processing power is increasing
  • Symbian OS makes programming on mobile phone possible
  • Develop motion input method on Symbian phone without requiring additional hardware
  • This acts as an innovative input method for applications such as:
    • Camera mouse to control the cursor
    • New input method for interactive games
    • Gesture input
why motion input
Why Motion Input?
  • Motion input is quick and convenient
  • Common way to play game on mobile phone is to use joystick or keypad
    • Limited freedom of movement
    • Lack of maneuverability
    • Difficult to press the small-sized keypad
  • Motion input doesn’t have these problems!
motion tracking
Motion Tracking
  • Motion tracking is a process to find the motion vector of the current frame from the reference frame
  • We use block matching algorithm to find the motion vector
block matching algorithm
Block Matching Algorithm

Comparison of


New Position

The motion vector which corresponds to the best match


block matching algorithm9
Block Matching Algorithm
  • Evaluate the "goodness" of a match by
    • Sum of Absolute Difference
  • Select the candidate block with the lowest error
motion tracking10
Motion Tracking
  • Block matching algorithm
    • Traditional block matching algorithm is slow
    • Our solution: Hybrid-type fast block matching algorithm
    • That is “Adaptive Spiral SEA PPNM SSD” algorithm
      • Adaptive Search Window
      • Spiral Scan Method
      • Successive Elimination Algorithm
      • Progressive Partial Norm Matching
      • Sum of Squared Difference
previous work
Previous Work
  • In the previous semester,
    • Developed a testing platform in both Window and Symbian OS
    • Developed our own block matching algorithm and feature selection algorithm
    • Built a real-time translational motion tracking engine
    • Developed some applications using the engine
improvement on feature selection
Improvement on Feature Selection
  • A good feature block:
    • High variance -> complex block
    • Not in a repeated pattern
  • If a good feature block is found, the performance of the motion tracking will be improved
  • We have made improvements to our existing algorithm
old feature selection


Old Feature Selection
  • Divide the current frame into squares
  • Apply Feature Selection Algorithm to each square
  • However, the best feature block may appear between two squares

Best feature block

new feature selection
New Feature Selection
  • More blocks are sampled
    • Sample 26x26 squares with size 15x15 in a 54x54 window
    • Old feature selection only samples 12x12 squares
  • Search in spiral way and stop searching when selection criteria are matched
    • Prefer to find a feature block in the center
    • Prevent out of bound problem (block appear out of screen)
new feature selection15
New Feature Selection
  • If a feature block is found on the edge, the following will happen:
    • The tracking algorithm will fail to find the exact match
  • Solution:
    • Apply additional constraint
new feature selection16
New Feature selection
  • In order to find a feature block that is not on the edge, we apply a checking algorithm
  • Important difference in all directions -> interest point
new feature selection17
New Feature Selection
  • Now our newest feature selection algorithm becomes:
    • Find a block with a large variance which indicates the complexity of the block
    • Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors
      • If either one of the SAD is small -> the block is on edge -> reject
      • Else the block is not on edge
rotation tracking engine
Rotation Tracking Engine
  • Observation and Motivation
    • As we rotate the phone, the object can still be tracked correctly (center of object always roughly equals center of green box)

Rotation Tracking Engine

  • First approach
    • Track one block by translational motion tracking engine, rotate the current block and then find which rotation of the current block best match with the previous block
    • Fail if tracking object is the same for different angle

One-block approach


Rotation Tracking Engine

  • Our approach
    • If two blocks are tracked simultaneously, angle of line connecting the two blocks indicates the rotation angle of the phone
    • Use only translational motion tracking

Two-block approach

rotation tracking engine21
Rotation Tracking Engine
  • Modification of adaptive search window method to suit rotation tracking
    • Phone’s motion can be both linear motion and circular motion
    • Linear Adaptive method used in translational motion tracking is not used in this engine
  • Why need adaptive search window method?
    • To increase performance based on the fact that during block matching, reaching the best candidate block earlier increases the elimination effect of SEA, PPNM, PDE
rotation tracking engine22
Rotation Tracking Engine
  • What to predict?
    • Predict the coordinates of the blocks in the next frame
    • Given the coordinates of the tracking blocks in the current frame and the previous frames
  • When to predict?
    • After each run of rotation tracking algorithm

Line L1

Line L2



Line L3

rotation tracking engine23
Rotation Tracking Engine
  • Solution
    • Angle between L2 and L3 = θ
      • θ = Tan-1(slope(L2)) – Tan-1(slope(L3))
    • Coordinates of the next tracking block (xL1, yL1) are calculated by multiplying the column matrix of coordinates of the previous block with a rotation matrix
rotation tracking engine24
Rotation Tracking Engine
  • Solution
    • The prediction of the positions of the next tracking blocks should also take the translational movement into account
    • Horizontal displacement = Tx
      • Tx = ( xL21 +xL22– xL31 -xL32 )/2
    • Vertical displacement = Ty
      • Ty = ( yL21 +yL22– yL31 -yL32 )/2
    • Coordinates of the next tracking block (xL1, yL1) is calculated by
rotation tracking engine25

Level 0

- 40°


Level 1

Level -1


Level 2


Level 3

Rotation Tracking Engine
  • Reducing error by using level
    • Increase/decrease one level only when change is large
    • To give less sensitive but more desirable output for game
    • e.g. skiing game: skier face only to 7 directions
    • Reduce difficulty of the game, increase reliability of the engine
rotation tracking engine26
Rotation Tracking Engine
  • Some screenshot showing how rotation tracking is carried out and how the result is displayed
rotation tracking engine27
Rotation Tracking Engine

A short video demo

tailor made background for both engines
Tailor-made background for both engines
  • Objective
    • Max. performance measurement
    • Increase usability
  • Condition
    • Feature selection can always find a good feature point
    • Within certain distance
      • No repeat pattern
      • Very distinct pattern
    • Pattern is nearly the same when rotated. E.g. Circle is good for rotation detection
virtual mouse
Virtual Mouse
  • An application that make full use of the translational motion tracking engine
  • Remote control the mouse of PC by mobile phone using motion as input
  • Advantages:
    • It allows input in many directions and provides high levels of control
    • Many buttons left for other purposes
      • E.g. Joystick can be used as course adjustment of mouse cursor
      • E.g. Some buttons can be used as shortcuts of applications
virtual mouse31
Virtual Mouse
  • Server
    • Configure Bluetooth device to provide RFComm Service and regards the Bluetooth transmission port as Comm. Port
    • Server receives message from that Comm. Port
    • Call function in MouseAction.h to make the mouse move and trigger mouse click event
virtual mouse32
Virtual Mouse
  • Client
    • Search for Bluetooth device nearby
    • Connect to the selected Bluetooth device
    • Every time motion tracking algorithm finishes, results are sent to server (14 times/sec)
    • Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
car racing game
Car Racing Game
  • A “Car Racing Game” is developed using the motion tracking engine.
  • Game lags. It is because:
    • The engine takes time to find the motion vector
    • CPU speed of the Symbian phones is low
  • Solutions:
    • Double Buffering
    • Direct Screen Access
skiing game
Skiing Game
  • We have developed a skiing game that makes use of the rotation tracking engine
    • Rotate the phone to control the angle of the skier
  • After using the rotation tracking engine:
    • The game becomes more interactive
    • The game has more degree of freedom for rotating the skier
skiing game36
Skiing Game
  • The process of using the rotation tracking engine is very simple
    • Create an instance of the tracking engine
    • Call the function of the engine to find the motion vector
    • Use the motion vector for the game logic
experimental result on symbian phone
Experimental Result on Symbian phone
  • Testing Environment
    • Platform
      • Symbian Phone Nokia 6600
    • Algorithm
      • Our final (hybrid-type) algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method)
    • Algorithm parameter
      • Block size         = 17 x 17 (pixels)
      • Search window size     = 16 x 16 (pixels)
    • Number of block to track in each run of algorithm = 1 block ONLY
experimental result on symbian phone40
Experimental Result on Symbian phone
  • Testing Result (avg time to run)
    • Final algorithm
      • 7ms
    • Partial Spiral algorithm
      • 22ms
    • Full Exhaustive Search algorithm (the most simplest one)
      • 55ms
  • We have developed:
    • The real-time motion tracking algorithm and feature selection method
    • The translational and rotational motion tracking engine on Symbian
    • A series of applications/games that make full use of our engines
  • Virtual Mouse
  • Act as an innovative input device for games
  • If we want to track the movement in 3D, we have to:
    • Use two cameras
    • Know the focal length of the cameras
    • Know the distance between two cameras
  • Depth can be recovered with two images and triangulation
  • Therefore, stereo has two steps
    • Finding matching points in the images
    • Then using them to compute depth.






  • Suppose we have found the matching point P. Now the step to compute the depth is as follow:

Note: y-axis is perpendicular to the slide

b: distance between two cameras

f: focal length of the cameras

  • If b is the distance between the two cameras, and f is the focal length of the cameras, we have
  • Use similar triangular, we can solve to get
  • In our project, only one camera is available in the Symbian phone
  • If we want to find the z-axis movement, we can find it by comparing the size of the capture image
    • Getting closer -> larger image
    • Getting farther -> smaller image
  • However, we have tried that using this algorithm is not accurate enough