Skip this Video
Download Presentation
By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM

Loading in 2 Seconds...

play fullscreen
1 / 52

By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM - PowerPoint PPT Presentation

  • Uploaded on

A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System. By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM. Co-authors: Prof. Mayez Al-Mouhamed Dr. S. M. Buhari Dr. Talal Al Kharobi. Outline .

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM' - sasha

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System


Syed Mohammed Shamsul Islam

Lecturer B and PhD student

Department of Computer Engineering KFUPM


  • Prof. Mayez Al-Mouhamed
  • Dr. S. M. Buhari
  • Dr. Talal Al Kharobi
  • Introduction
  • Design Issues
  • Implementation
  • Performance Evaluation
  • Conclusions & Recommendations
  • A scheme that allows humans to extend their manipulative skills over a network.

Visual feedback

Visual feedback

Force feedback

Force feedback

telerobotic applications
Telerobotic Applications
  • Hazardous situations
    • e.g. bomb disposal
  • Scaled-down and scaled-up situations
    • e.g. micro-surgery
  • Situations affected by human presence
    • e.g. closed-chest heart bypass
  • Teaching, training, maintenance, entertainment.
problem of conventional tele robotics
Problem of Conventional Tele-robotics
  • Time delay:
      • Processing, copying video data, network delays
      • Move-n-Wait strategy
  • Operator has to follow trial and error method to perform a task, which might be dangerous for example in telesurgery.
improving performance by using ar
Improving Performance by Using AR
  • By overlaying virtual image with the real image operator e.g. surgeon can make a plan before going for actual teleoperation  safety.
  • Rehearsal and correction can be made in the simulation plan.
  • Saving bandwidth by sending (less frequently) only the planned data.
challenges of ar
Bi-directional one-to-one mapping between graphic and real co-ordinate spaces








Challenges of AR
  • Graphical robot arm must mach with real robot arm in the video.
challenges of ar1
Challenges of AR
  • Camera Calibration:
    • Finding accurate camera calibration is challenging.
  • Viewpoint Registration
design methodology
Performance evaluation of the approach


Superimposition of the virtual object on to the stereo-video and augmented tele-manipulation

Augmentation of


Interfacing to the telerobotic stereo-vision system


Moving capability of the virtual objects


Displaying the virtual objects

Virtual object rendering

Drawing the virtual objects

Virtual object drawing

Deriving the robot model equations

Mathematical models

Design Methodology
mathematical model of the robot manipulator
Mathematical Model of the Robot Manipulator
  • PUMA 560 Slave Arm:
    • 6 DOF
    • 6 links are interconnected

serially to each other except

first and last.

    • All the joints are rotational.

A rotational joint

mathematical model of the robot manipulator1
Each link of the arm is attached with a frame of reference Ri (xi,yi,zi) to specify its position and orientation.

Link Li+1 rotates w.r.t Li when frame Ri+1 rotates w.r.t either axes of xi, yi, or zi

Link vector OiOi+1,i can be expressed as


Mathematical Model of the Robot Manipulator
  • Now position and orientation of Oi+1 w.r.t Oi-1 can be expressed as
mathematical model of the robot arm
Where,Mathematical Model of the Robot Arm
  • Position and orientation of the end effector is expressed relative to the frame of the base.
  • So the geometric model of robot arm can be expressed as:

This model will give us a skeleton of the graphical arm.

building body shapes around the skeleton of graphical arm
Building Body Shapes Around the Skeleton of Graphical Arm
  • Alternatives:
    • Taking as is: link 3 and 4 are trapezoidal, others are cylindrical.
    • Considering all as cylindrical
building body shapes around the skeleton of graphical arm1
Building Body Shapes Around the Skeleton of Graphical Arm
  • Cylinder is drawn with finite element representation.
  • Increasing number of segments will improve quality of view but increase computational complexity.
  • Alternatives of primitive for drawing a cylinder:
    • Triangles:
      • Triangle stripe, Triangle fan, Triangle list
    • Lines
      • Line stripe, Line list
data structure design
Data Structure Design
  • Types of data and nature of manipulation:
    • A link is represented by its start and end point co-ordinates, radius, axis of rotation, joint angle, orientation matrix etc.
    • Each link is also associated with the vertices of its body shape.
    • Links are serially connected and position and orientation of a link depends on position and orientation of its previous link.
    • There is frequent movement of links in teleoperation and each movement involves matrix calculation for new position and orientation of the links to be changed.
  • Objectives:
    • To reduce processing time by reducing computation.
    • To provide flexibility and generality
data structure design1
Data Structure Design
  • Our strategy:
    • Each link is defined as an object geometric data of each link is now apart from the other link,.
    • Configuration data (number of links, number of segments in cylinder etc) kept in separate data file.
    • World and view transformation matrices are combined to reduce matrix multiplication.
    • Vertex data of body shape of each link is stored in separate vertex buffer.
displaying the graphical arm
Displaying the Graphical Arm
  • Given the robot structure in its model space
  • Rendering on the display screen:
    • Scene layout setup:
      • Transformation matirces, Camera position, target position, viewing angle, lighting.
    • Viewing Models
      • Solid Model or Wire-frame Model
      • Psychological studies and one performance studies in telerobotic system of shows no significant diff.
      • Wire-frame models have some adv.-reduce occlusion, lesser rendering time.
    • Rendering:
      • Scanline rendering, Z-buffering algorithm for HSR.
sample display of graphical arm
Sample Display of Graphical Arm

Wire-frame view

Solid view

movement of graphical arm in cartesian space
Movement of Graphical Arm in Cartesian Space


Receive the incremental position (X)

and orientation (M) matrices

Calculate new position, Xnew(t)

and orientation, Mnew(t) matrices

based on current frame of reference

Calculate corresponding angular values

Using Inverse Geometric Transformation

(new=G-1(Xnew, Mnew)

Angle within



acquisition of real video image
Acquisition of Real Video Image
  • The MDTF Approach is used for stereovision.
    • It is multi-threaded, distributed and provide better performance in video transfer.
  • Real video image of the slave robot at the server side is captured simultaneously by two video cameras.
  • Then a reliable client-server connection is established and upon a request from the client a stereo frame comprising of two pictures is sent over LAN through window sockets.
  • On the client side after detecting and making connection with the server pictures are received and displayed on the screen and on the HMD when connected.
  • A double buffer, concurrent transfer approach is used to maximize overlapped transfer activities between cameras, processor and the network.
3d visualization tools and techniques
3D Visualization Tools and Techniques
  • Technique
    • Alternatives: Sync-Doubling, Page-flipping
    • Chosen: Page-flipping
      • Provides higher resolution and avoid ”flashing” problem of 3D imaging.
  • Display device
    • Alternatives: HMD, monitor, eye-shuttering glass
    • Chosen: HMD, monitor
      • No flickering, easy to work
3d visualization example
3D Visualization Example

A snap shot of a stereo-image.

camera calibration
Camera Calibration
  • Heikkila’s calibration method which provides very accurate results. It shows accuracy up to 1/50 of the pixel size.
  • Pinhole camera model with perspective projection and least-square error optimization.
  • The intrinsic camera parameters:
    • Focal length, fc
    • Principal point, cc
    • Skew co-efficient, alpha_c
    • Lens distortion co-efficient, kc
camera calibration1
Camera Calibration…

The aspect ratio= fc(2) / fc(1)

The field of view angle can be calculated from,

camera calibration2
Camera Calibration…
  • Extrinsic Parameters

If XX=[X,Y,Z] and XXc are co-ordinate of P in grid and camera ref, then

Where, translation vector Tc_1 is the co-ordinate vector of O in camera ref frame and the rotation matrix Rc_1 is the surface normal vector of the grid plane in the camera ref frame.

3d visualization algorithm
3D Visualization Algorithm
  • Step 1: Acquire video image and copy this memory stream to a surface( say FrontSurf).
  • Step 2: Copy the FrontSurf surface to another temporary surface (say, AugSurf).
  • Step 3: Draw graphical objects/change in graphical objects on AugSurf.
  • Step 4: Copy AugSurf to the surface that will be used to display (say, backSurf).
  • Step 5: Display backSurf with left camera image to left view port/monitor.
  • Step 6: Display backSurf with right image to right view port/monitor.
  • Step 7: Observe 3D with HMD.
graphical tele manipulation
Graphical Tele-manipulation
  • How Graphic Manipulation Corresponds to the Real
    • If the base link of the real robot remains fixed relative to the video cameras, the base link of the graphical arm will also remain fixed relative to the graphical cameras.
    • The end-effector of the graphical arm can be manipulated in the graphical coordinate space, relative to objects in the task space (keeping base link in same location of real robot base).
gui design
GUI Design
  • Client Side Input/Output Specification
      • Connecting to the server PC, master arm and HMD
      • Receiving and displaying the stereo video.
      • Taking user's input for movement of the real and graphical robot for simulation
  • Means of User Interaction:
      • Master arm
      • Joystick
      • Keypad
      • Mouse
tools used
Tools Used
  • Client-server communication platform
    • MS .NET with .NET Remoting
  • Programming Language
    • Visual C#.NET
  • Graphics Tool
    • Microsoft DirectX
  • 3D Graphics API
    • Alternatives:
      • Windows GDI
      • Java3D by Sun MicroSystems
      • Open GL (Open Graphics Language) by SGI Silicon Graphics
      • Direct3D
motivation for using direct3d
Motivation for Using Direct3D
  • It increases performance by using hardware acceleration.
  • It allows applications to run full-screen instead of embedded in a window. As we are using HMD this feature is very helpful in our case to have the 3D effect.
  • Direct3D perfectly match with Microsoft's various Windows operating systems and with Microsoft .NET framework which are used in our telerobotic client-server framework
graphics implementation
Graphics Implementation
  • Virtual Object Modeling Module
    • Defines the structure of the virtual objects (robot, cubes etc.)
    • Handles functions like movement of virtual objects.
  • Display Module:
    • Synchronization of real and virtual data
    • Projection on video surface
    • Augmentation of real video
    • Page Flipping for HMD stereo visualization
evaluation of graphical system
Evaluation of Graphical System:
  • Refresh Rate:

[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

time required for rendering graphical arm
Time Required for Rendering Graphical Arm

[Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

  • Re-projection error with the calibration method:

[Pixel Error (0.11689,0.11500)]

Pixel Axis in the y direction

Pixel Axis in the x direction

comparison to other apporaches
Comparison to Other Apporaches
  • Iqbal, A. [1] augmented with only a small red ball at the position of gripper in comparison to our whole graphical arm.
  • Iqbal, A. [1] used Faugeras [4] calibration with Kuno[5]’s affine frame of reference which led him to noticeable mismatch with real error in the matching shown in his figure. Whereas our computer vision-based calibration reduces error upto 1/50 of pixel size.
  • J. Vallino reports in his PhD thesis refresh rate of 10fps to be required for AR, when we gets above 11-17fps after overlaying graphical arm with live stereo video.
  • Graphics rendering of our system is faster than Iqbal, A. [1],[2][3] for our use of Direct3D.
  • Our system is comparatively cheaper due to the use of commodity hardware (PC) and software.
summary of the work and contributions
Summary of the Work and Contributions
  • Using hardware accelerated graphics rendition that provides us with excellent refresh rate of the output screen.
  • Improvement in accuracy of execution by using better calibration method and graphic aids (accuracy up to 1/50 of pixel size).
  • User-friendly graphical user interface for simple manipulation in the telerobotic AR system.
summary of the work and contributions1
Summary of the Work and Contributions…
  • Flexible and generalized data structure suitable for telerobotic visualization.
  • Identifying design strategy for intelligent switching in VR and AR mode for ensuring QoS.
  • Use of cheap and commercially available hardware and software
  • Can be used as a cheap and flexible visual tool for showing robot manipulation in the classrooms.
future research directions
Future Research Directions
  • Providing an intelligent system to switch between VR and AR modes of operation based on network delays to ensure QoS.
  • Using multi-processor system for processing video and graphics data more efficiently.
  • Using commercial software available to extract exact 3D model of the workspace objects which will facilitate more accurate task manipulation.
  • Iqbal, A. Multistream realtime control of a distributed telerobotic system. M.Sc. Thesis, King Fahd University of Petroeum and Minerals,June 2003.
  • A Rastogi, P. Milgram, and D. Drascic. Telerobotic control with stereoscopic augmented reality. SPIE, Vol.2653: Stereoscopic Displays and Virtual Reality Systems III:135{146, Feb. 1996.
  • R. Marin, P.J. Sanz, and J.S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. Proc. IEEEInternational Conference on Robotics and Automation, 2002 ICRA '02, Vol.3:2725 - 2730, May 2002.
  • O.D. Faugeras and G. Toscani. The calibration problem for stereo. Proceedings of Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, Vol. 5, No. 3, June, 15-20 1986.

[5]Y. Kuno, K. Hayashi, K.H. Jo, and Y. Shirai. Human-robot interface using uncalibrated stereo vision. International Conference on Intelligent Robots and Systems 95, 1:525{530, 1995.

[6]J. Abdullah and K. Martinez. Camera self-calibration for the ar-toolkit. The First IEEE International Workshop on Augmented Reality Toolkit, page 5, Sept. 2002.

[7]J.-Y. Herve, C. Duchesne, and V. Pradines. Dynamic registration for augmented reality in telerobotics applications. IEEE International Conference on Systems, Man, and Cyberneticsn, Vol.2:1348-1353, October 2000.

Thank You

Questions are welcome