1 / 21

Cpre 185 Wii/Kinect Lab Software

Cpre 185 Wii/Kinect Lab Software. Senior Design Group 16. Client / Adviser : Dr. Tom Daniels Brenton Hankins, Rick Hanton , Harsh Goel , Jeff Kramer. Problem Statement.

lundy
Download Presentation

Cpre 185 Wii/Kinect Lab Software

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Cpre 185 Wii/Kinect Lab Software Senior Design Group 16 Client / Adviser: Dr. Tom Daniels BrentonHankins, Rick Hanton, Harsh Goel, Jeff Kramer

  2. Problem Statement • Freshmen computer engineering college students and high school students need an interesting way to be introduced to programming and problem solving. • In the past, Iowa State students have been taught to create simple programs in the C programming language that use Nintendo Wiimotes as input devices. • Software needs to be developed similar to Wii Wrap, but uses the Microsoft Kinect™ sensor as input to student-written programs.

  3. Hardware - Kinect • RGB Webcam Camera 640x480 @ 30 Hz (center) • IR Projector Projects Grid for Depth Mapping • IR Camera at 640x480 @ 30 Hz collects depth data (right) • Microphones give room audio • Small Panning motor turns sensor bar from side to side • Onboard image preprocessing • Projector power draw requires extra AC power

  4. Hardware - Wii • Wii Remote / Wiimote • 11 buttons + Power • Roll/Pitch/Yaw Gyro • Infrared Sensor • Speaker (unsupported) • Optional Nunchuck

  5. Schedule / Cost

  6. Architecture • Three Modules: • Hardware interface to read in raw data (provided) • Interface simplification / data processing • Graphical interface to display output

  7. Flow Diagram

  8. WiiMote Solutions • Issue: Decrease delay in printf output • Solution: Added a fflush(stdout) statement immediately after printf • Issue: IR data becomes erratic if sensor loses source for one frame of data • Solution: Save last good frame of IR data (x,y) and loop the last good data until new data arrives

  9. Kinect Design Approaches • OpenKinect • Tricky to get started with in windows • Natively supports Unix • Body tracking not built-in • Needs a calibration pose for body tracking • No event when new Video or Depth frame is available • Microsoft Kinect SDK • Sample Projects • Documentation • Simple installer • Natively supports Windows • Full body tracking built-in

  10. Microsoft Kinect SDK • Data Available • RGB Video Frames(640x480 max resolution) • Depth (mm) Data Frames (320x240 max resolution) • Skeletal tracking points (20 points per player) • How we get the data • Pass an event object to alert us when new data is available • Pass a video/depth/skeleton frame pointer to read in next frame • Run event loop and process each frame as RGB/Depth/Skeleton data

  11. Shared Memory Use • Windows File Mapping is an inter-process communication tool that maps a block of memory by name • Shared Memory in Kinect Wrap • File Mapping is set up by Kinect Wrap backend program with a specific name for each block • Backend gets data from Kinect SDK and dumps it into shared memory space when each new frame becomes available • Our API Library copies frame from shared memory into local memory space for student program to use.

  12. Kinect Wrap API • Provides an easy-to-use interface for students while maximizing functionality • Functions: • getVideoFrame(rgb** pFrame) • getDepthFrame(short** pFrame) • getSkeleton(Point3D* pSkel) • setOutputDisplay(rgb** pFrame) • setOutputText(char* sText) • setPixel(rgb** pFrame, int x, int y, KCOLOR color)

  13. Kinect Wrap Library • Kinect Wrap Library is a static Windows library that students must include their C projects • Library handles backend processing of API • Abstracts all of the technical workings of the Kinect SDK, File Mapping API, and GUI design

  14. Kinect GUI • Abstracts away the details of drawing Kinect data via a graphical API • Kinect GUI receives data via shared memory • Data is drawn as a grid of vertices, each with a custom RGB value via DirectX • Intend to port GUI code to OpenGL and work on constructing a more intuitive GUI in future

  15. Testing – Wiimotes • Field Testing • Compared time delay of data transmission between the current and previous iterations of the Wii Wrap code. • Found the apparent delay seen in the Wii Wrap GUI to be significantly decreased. • Test results corroborated by CPRE 185 lab TAs during classroom use. • No previous functionality was affected by these changes.

  16. Kinect Black Box Testing • Assumed that the Kinect SDK functions deliver correct data from Kinect sensor • Shared Memory • Wrote known data into shared memory space with one program and had second program read it out and verify that the data has not been altered • Student API Interface • Verified that buffers of known data sent to the student program are not altered before being read • Verified that a known function run by student code produces the expected output frame data given a known input frame

  17. Kinect Black Box Testing • GUI Output • Tested whether frames were being updated consistently by writing specific colors to known pixels in a given output frame

  18. Kinect Integration Testing • setPixel() • Tested for boundary values. • Attempted to set pixels outside the frame (640x480) • Attempted to set pixel RGB values outside of 0-255 • getVideoFrame() & setOutputDisplay() • Check that output frame RGB data matches input data • Found that with simple student processing code, any reduction in output fps from device frame rate is minimal • getDepthFrame() • Used a plane at a known distance from the Kinect to find how accurate the depth values in mm are vs. reality.

  19. Kinect Lab Example intmain() { KinectLib_Init(); // Initialize the library rgb* frame[VIDEO_HEIGHT]; // Create 2D frame of rgb data short* dframe[DEPTH_HEIGHT]; // Create 2D frame of short depth data Point3D* skeleton; // Create array of skeleton data while(1){ getVideoFrame(frame); // Map video frame to pointers getDepthFrame(dframe); // Map depth frame to pointers if(skeletonTracked()) { skeleton = getSkeleton(); // Map Skeleton data } setPixel(frame, 23, 18, BLUE); // Arbitrarily set pixel (23, 18) to Blue displayVideoFrame(frame); // Display the modified video frame in the GUI } KinectLib_Destroy(); // Destroy Kinect library instance return 1; }

  20. Future Work • Multi-Kinect Support • More comprehensive GUI program • Additional Library Functionality • Create code to share Kinect data over Ethernet

  21. Questions? seniord.ece.iastate.edu/dec1116/

More Related