Robotic application of
1 / 11

Robotic application of - PowerPoint PPT Presentation

  • Uploaded on

Robotic application of. Human Hand Gesture. S. Ali El-Gabri, Al- Noor Academy Nathaniel Mahowald , Mass Academy Grad Students: Dimitri Kanoulas and Andreas Ten Pas PI: Robert Platt. Introduction. The fundamental point of this project is:

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Robotic application of' - cale

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Robotic application of

Human Hand Gesture


Ali El-Gabri, Al-Noor Academy

Nathaniel Mahowald, Mass Academy

Grad Students: DimitriKanoulas and Andreas Ten Pas

PI: Robert Platt


  • The fundamental point of this project is:

    • Making gestures that make the robot pick up objects

    • Pointing at the object the user wants the Robot to pick up.

  • How?

    • Creating interfaces between the computer and the sensor

    • Creating interfaces between the sensor and Robot.


  • The materials used were:

    • ROS Hydro Medusa

      • Robot Operating System

      • Provides several helpful tools

      • Hydro Medusa 7th Version

    • Xtion Pro

      • Like a Kinnect

      • Makes gesture tracking precise

    • Baxter

      • Robot – two hand manipulator


  • Install ROS Hydro Medusa

  • Install openni_launch

    • Camera driver

  • Install openni_tracker

    • Creates a skeleton for any person in front of it

  • Set up a catkin workspace

  • Write python code to communicate

Sub-project 1: Directional Pointer

  • Work with Rviz

    • ROS Visualization; visualizes camera feed

  • Set up the Transforms (TF) in Rviz

    • Keeps track of 3D frames’ change over time

    • Operates in a distributed system

  • Created a TF listener in python

    • Receives coordinate frames

    • Query for specific TFs between frames

  • Functional code

    • Informs user of the direction arm is in the xyz plane

Sub-project 2: Body Part Pointer

  • Have two users on screen

  • Point with left hand

    • Display body part being pointed at for either user

    • Display which user is pointing at which

  • How this helps:

    • More work with Rviz

    • Rviz already recognized human bodies

    • Experimented with Dot Product, Matrices, and creation of Vectors

    • First step to pointing at other things

Sub-project 3: Gesture Control

  • Method of gesture based control without fixed frame

  • This is the first place where we fixed our user buildup problem

  • Went through a few drafts of what positions worked

  • First project we worked with on the robot

Final Project: Any Frame Pointer

  • We couldn’t get a “true” pointer without creating a fixed frame; our solution was calibration

  • Extremely accurate

  • Uses left hand as a signal that user is pointing

    • Potential extensions

Next Steps

  • Use voice recognition software to interact with the user

  • Create a pointer that does not require calibration and uses frame to run

  • Compile all the code onto a usable device, so that a disabled person could use a robotic arm to pick up objects they need


  • A special thanks to our very helpful Grad students, DimitriKanoulas and Andreas Ten Pas.

  • A very warm appreciation to Robert Platt, our ever wise PI.

  • And, of course, to those who made it possible and walked us every step of the way:

  • Duggan Claire, Program Director

  • Kassi Stein, Program Coordinator

  • Chi-Yin Tse, Program Coordinator