A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision Sys...
1 / 52

By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM - PowerPoint PPT Presentation

  • Uploaded on

A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System. By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM. Co-authors: Prof. Mayez Al-Mouhamed Dr. S. M. Buhari Dr. Talal Al Kharobi. Outline .

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' By Syed Mohammed Shamsul Islam Lecturer B and PhD student Department of Computer Engineering KFUPM' - sasha

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

A Hierarchical Design Scheme for Application of Augmented Reality in a Telerobotic Stereo-Vision System


Syed Mohammed Shamsul Islam

Lecturer B and PhD student

Department of Computer Engineering KFUPM


  • Prof. Mayez Al-Mouhamed

  • Dr. S. M. Buhari

  • Dr. Talal Al Kharobi

Outline Reality in a Telerobotic Stereo-Vision System

  • Introduction

  • Design Issues

  • Implementation

  • Performance Evaluation

  • Conclusions & Recommendations

Telerobotics: Reality in a Telerobotic Stereo-Vision System

  • A scheme that allows humans to extend their manipulative skills over a network.

Visual feedback

Visual feedback

Force feedback

Force feedback

Telerobotic applications
Telerobotic Applications Reality in a Telerobotic Stereo-Vision System

  • Hazardous situations

    • e.g. bomb disposal

  • Scaled-down and scaled-up situations

    • e.g. micro-surgery

  • Situations affected by human presence

    • e.g. closed-chest heart bypass

  • Teaching, training, maintenance, entertainment.

Problem of conventional tele robotics
Problem of Conventional Tele-robotics Reality in a Telerobotic Stereo-Vision System

  • Time delay:

    • Processing, copying video data, network delays

    • Move-n-Wait strategy

  • Operator has to follow trial and error method to perform a task, which might be dangerous for example in telesurgery.

  • Improving performance by using ar
    Improving Performance by Using AR Reality in a Telerobotic Stereo-Vision System

    • By overlaying virtual image with the real image operator e.g. surgeon can make a plan before going for actual teleoperation  safety.

    • Rehearsal and correction can be made in the simulation plan.

    • Saving bandwidth by sending (less frequently) only the planned data.

    Challenges of ar

    Bi-directional one-to-one mapping between graphic and real co-ordinate spaces








    Challenges of AR

    • Graphical robot arm must mach with real robot arm in the video.

    Challenges of ar1
    Challenges of AR co-ordinate spaces

    • Camera Calibration:

      • Finding accurate camera calibration is challenging.

    • Viewpoint Registration

    Design Issues co-ordinate spaces

    Design methodology

    Performance evaluation of the approach co-ordinate spaces


    Superimposition of the virtual object on to the stereo-video and augmented tele-manipulation

    Augmentation of


    Interfacing to the telerobotic stereo-vision system


    Moving capability of the virtual objects


    Displaying the virtual objects

    Virtual object rendering

    Drawing the virtual objects

    Virtual object drawing

    Deriving the robot model equations

    Mathematical models

    Design Methodology

    Mathematical model of the robot manipulator
    Mathematical Model of the Robot Manipulator co-ordinate spaces

    • PUMA 560 Slave Arm:

      • 6 DOF

      • 6 links are interconnected

        serially to each other except

        first and last.

      • All the joints are rotational.

    A rotational joint

    Mathematical model of the robot manipulator1

    Each link of the arm is attached with a frame of reference Ri (xi,yi,zi) to specify its position and orientation.

    Link Li+1 rotates w.r.t Li when frame Ri+1 rotates w.r.t either axes of xi, yi, or zi

    Link vector OiOi+1,i can be expressed as


    Mathematical Model of the Robot Manipulator

    • Now position and orientation of Oi+1 w.r.t Oi-1 can be expressed as

    Mathematical model of the robot arm

    Where, Ri (x

    Mathematical Model of the Robot Arm

    • Position and orientation of the end effector is expressed relative to the frame of the base.

    • So the geometric model of robot arm can be expressed as:

    This model will give us a skeleton of the graphical arm.

    Building body shapes around the skeleton of graphical arm
    Building Body Shapes Around the Skeleton of Graphical Arm Ri (x

    • Alternatives:

      • Taking as is: link 3 and 4 are trapezoidal, others are cylindrical.

      • Considering all as cylindrical

    Building body shapes around the skeleton of graphical arm1
    Building Body Shapes Around the Skeleton of Graphical Arm Ri (x

    • Cylinder is drawn with finite element representation.

    • Increasing number of segments will improve quality of view but increase computational complexity.

    • Alternatives of primitive for drawing a cylinder:

      • Triangles:

        • Triangle stripe, Triangle fan, Triangle list

      • Lines

        • Line stripe, Line list

    Data structure design
    Data Structure Design Ri (x

    • Types of data and nature of manipulation:

      • A link is represented by its start and end point co-ordinates, radius, axis of rotation, joint angle, orientation matrix etc.

      • Each link is also associated with the vertices of its body shape.

      • Links are serially connected and position and orientation of a link depends on position and orientation of its previous link.

      • There is frequent movement of links in teleoperation and each movement involves matrix calculation for new position and orientation of the links to be changed.

    • Objectives:

      • To reduce processing time by reducing computation.

      • To provide flexibility and generality

    Data structure design1
    Data Structure Design Ri (x

    • Our strategy:

      • Each link is defined as an object geometric data of each link is now apart from the other link,.

      • Configuration data (number of links, number of segments in cylinder etc) kept in separate data file.

      • World and view transformation matrices are combined to reduce matrix multiplication.

      • Vertex data of body shape of each link is stored in separate vertex buffer.

    Displaying the graphical arm
    Displaying the Graphical Arm Ri (x

    • Given the robot structure in its model space

    • Rendering on the display screen:

      • Scene layout setup:

        • Transformation matirces, Camera position, target position, viewing angle, lighting.

      • Viewing Models

        • Solid Model or Wire-frame Model

        • Psychological studies and one performance studies in telerobotic system of shows no significant diff.

        • Wire-frame models have some adv.-reduce occlusion, lesser rendering time.

      • Rendering:

        • Scanline rendering, Z-buffering algorithm for HSR.

    Sample display of graphical arm
    Sample Display of Graphical Arm Ri (x

    Wire-frame view

    Solid view

    Movement of graphical arm in cartesian space
    Movement of Graphical Arm in Cartesian Space Ri (x


    Receive the incremental position (X)

    and orientation (M) matrices

    Calculate new position, Xnew(t)

    and orientation, Mnew(t) matrices

    based on current frame of reference

    Calculate corresponding angular values

    Using Inverse Geometric Transformation

    (new=G-1(Xnew, Mnew)

    Angle within



    Acquisition of real video image
    Acquisition of Real Video Image Ri (x

    • The MDTF Approach is used for stereovision.

      • It is multi-threaded, distributed and provide better performance in video transfer.

    • Real video image of the slave robot at the server side is captured simultaneously by two video cameras.

    • Then a reliable client-server connection is established and upon a request from the client a stereo frame comprising of two pictures is sent over LAN through window sockets.

    • On the client side after detecting and making connection with the server pictures are received and displayed on the screen and on the HMD when connected.

    • A double buffer, concurrent transfer approach is used to maximize overlapped transfer activities between cameras, processor and the network.

    3d visualization tools and techniques
    3D Visualization Tools and Techniques Ri (x

    • Technique

      • Alternatives: Sync-Doubling, Page-flipping

      • Chosen: Page-flipping

        • Provides higher resolution and avoid ”flashing” problem of 3D imaging.

    • Display device

      • Alternatives: HMD, monitor, eye-shuttering glass

      • Chosen: HMD, monitor

        • No flickering, easy to work

    3d visualization example
    3D Visualization Example Ri (x

    A snap shot of a stereo-image.

    Camera calibration
    Camera Calibration Ri (x

    • Heikkila’s calibration method which provides very accurate results. It shows accuracy up to 1/50 of the pixel size.

    • Pinhole camera model with perspective projection and least-square error optimization.

    • The intrinsic camera parameters:

      • Focal length, fc

      • Principal point, cc

      • Skew co-efficient, alpha_c

      • Lens distortion co-efficient, kc

    Camera calibration1
    Camera Calibration… Ri (x

    The aspect ratio= fc(2) / fc(1)

    The field of view angle can be calculated from,

    Camera calibration2
    Camera Calibration… Ri (x

    • Extrinsic Parameters

    If XX=[X,Y,Z] and XXc are co-ordinate of P in grid and camera ref, then

    Where, translation vector Tc_1 is the co-ordinate vector of O in camera ref frame and the rotation matrix Rc_1 is the surface normal vector of the grid plane in the camera ref frame.

    3d visualization algorithm
    3D Visualization Algorithm Ri (x

    • Step 1: Acquire video image and copy this memory stream to a surface( say FrontSurf).

    • Step 2: Copy the FrontSurf surface to another temporary surface (say, AugSurf).

    • Step 3: Draw graphical objects/change in graphical objects on AugSurf.

    • Step 4: Copy AugSurf to the surface that will be used to display (say, backSurf).

    • Step 5: Display backSurf with left camera image to left view port/monitor.

    • Step 6: Display backSurf with right image to right view port/monitor.

    • Step 7: Observe 3D with HMD.

    Graphical tele manipulation
    Graphical Tele-manipulation Ri (x

    • How Graphic Manipulation Corresponds to the Real

      • If the base link of the real robot remains fixed relative to the video cameras, the base link of the graphical arm will also remain fixed relative to the graphical cameras.

      • The end-effector of the graphical arm can be manipulated in the graphical coordinate space, relative to objects in the task space (keeping base link in same location of real robot base).

    Gui design
    GUI Design Ri (x

    • Client Side Input/Output Specification

      • Connecting to the server PC, master arm and HMD

      • Receiving and displaying the stereo video.

      • Taking user's input for movement of the real and graphical robot for simulation

  • Means of User Interaction:

    • Master arm

    • Joystick

    • Keypad

    • Mouse

  • Tools used
    Tools Used Ri (x

    • Client-server communication platform

      • MS .NET with .NET Remoting

    • Programming Language

      • Visual C#.NET

    • Graphics Tool

      • Microsoft DirectX

    • 3D Graphics API

      • Alternatives:

        • Windows GDI

        • Java3D by Sun MicroSystems

        • Open GL (Open Graphics Language) by SGI Silicon Graphics

        • Direct3D

    Motivation for using direct3d
    Motivation for Using Direct3D Ri (x

    • It increases performance by using hardware acceleration.

    • It allows applications to run full-screen instead of embedded in a window. As we are using HMD this feature is very helpful in our case to have the 3D effect.

    • Direct3D perfectly match with Microsoft's various Windows operating systems and with Microsoft .NET framework which are used in our telerobotic client-server framework

    Graphics implementation
    Graphics Implementation Ri (x

    • Virtual Object Modeling Module

      • Defines the structure of the virtual objects (robot, cubes etc.)

      • Handles functions like movement of virtual objects.

    • Display Module:

      • Synchronization of real and virtual data

      • Projection on video surface

      • Augmentation of real video

      • Page Flipping for HMD stereo visualization

    Gui implementation
    GUI Implementation Ri (x

    • Main Form

    Gui implementation1
    GUI Implementation Ri (x

    • Stereo Form

    Evaluation of graphical system
    Evaluation of Graphical System: Ri (x

    • Refresh Rate:

    [Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

    Time required for rendering graphical arm
    Time Required for Rendering Graphical Arm Ri (x

    [Average is taken over 1000 samples, running on Pentium-4, 2GHz machine with 1GB RAM]

    Accuracy Ri (x

    • Re-projection error with the calibration method:

      [Pixel Error (0.11689,0.11500)]

    Pixel Axis in the y direction

    Pixel Axis in the x direction

    Comparison to other apporaches
    Comparison to Other Apporaches Ri (x

    • Iqbal, A. [1] augmented with only a small red ball at the position of gripper in comparison to our whole graphical arm.

    • Iqbal, A. [1] used Faugeras [4] calibration with Kuno[5]’s affine frame of reference which led him to noticeable mismatch with real error in the matching shown in his figure. Whereas our computer vision-based calibration reduces error upto 1/50 of pixel size.

    • J. Vallino reports in his PhD thesis refresh rate of 10fps to be required for AR, when we gets above 11-17fps after overlaying graphical arm with live stereo video.

    • Graphics rendering of our system is faster than Iqbal, A. [1],[2][3] for our use of Direct3D.

    • Our system is comparatively cheaper due to the use of commodity hardware (PC) and software.

    Conclusion Ri (x

    Summary of the work and contributions
    Summary of the Work and Contributions Ri (x

    • Using hardware accelerated graphics rendition that provides us with excellent refresh rate of the output screen.

    • Improvement in accuracy of execution by using better calibration method and graphic aids (accuracy up to 1/50 of pixel size).

    • User-friendly graphical user interface for simple manipulation in the telerobotic AR system.

    Summary of the work and contributions1
    Summary of the Work and Contributions… Ri (x

    • Flexible and generalized data structure suitable for telerobotic visualization.

    • Identifying design strategy for intelligent switching in VR and AR mode for ensuring QoS.

    • Use of cheap and commercially available hardware and software

    • Can be used as a cheap and flexible visual tool for showing robot manipulation in the classrooms.

    Future research directions
    Future Research Directions Ri (x

    • Providing an intelligent system to switch between VR and AR modes of operation based on network delays to ensure QoS.

    • Using multi-processor system for processing video and graphics data more efficiently.

    • Using commercial software available to extract exact 3D model of the workspace objects which will facilitate more accurate task manipulation.

    References Ri (x

    • Iqbal, A. Multistream realtime control of a distributed telerobotic system. M.Sc. Thesis, King Fahd University of Petroeum and Minerals,June 2003.

    • A Rastogi, P. Milgram, and D. Drascic. Telerobotic control with stereoscopic augmented reality. SPIE, Vol.2653: Stereoscopic Displays and Virtual Reality Systems III:135{146, Feb. 1996.

    • R. Marin, P.J. Sanz, and J.S. Sanchez. A very high level interface to teleoperate a robot via web including augmented reality. Proc. IEEEInternational Conference on Robotics and Automation, 2002 ICRA '02, Vol.3:2725 - 2730, May 2002.

    • O.D. Faugeras and G. Toscani. The calibration problem for stereo. Proceedings of Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, Vol. 5, No. 3, June, 15-20 1986.

    References Ri (x

    [5]Y. Kuno, K. Hayashi, K.H. Jo, and Y. Shirai. Human-robot interface using uncalibrated stereo vision. International Conference on Intelligent Robots and Systems 95, 1:525{530, 1995.

    [6]J. Abdullah and K. Martinez. Camera self-calibration for the ar-toolkit. The First IEEE International Workshop on Augmented Reality Toolkit, page 5, Sept. 2002.

    [7]J.-Y. Herve, C. Duchesne, and V. Pradines. Dynamic registration for augmented reality in telerobotics applications. IEEE International Conference on Systems, Man, and Cyberneticsn, Vol.2:1348-1353, October 2000.

    Thank You Ri (x

    Questions are welcome