Skeleton Tracking in Near Mode: Enabling and Monitoring Changes in the Skeleton

1. EEC-492/592Kinect Application Development Lecture 9 Wenbing Zhao wenbing@ieee.org

2. Outline Human skeleton tracking (part II)

3. Skeleton Tracking in Near Mode • Must enable near mode for DepthStream and near mode tracking for SkeletonStream • By default EnableTrackingInNearRange is set to false this.sensor.DepthStream.Range = DepthRange.Near; this.sensor.SkeletonStream.EnableTrackingInNearRange = true;

4. The Skeleton Class

5. Choosing Which Skeleton to Track • Each skeleton is identified by a unique integer identifer: TrackingId • The TrackingId for a tracked skeleton is positive (>0) • The TrackingId of the skeleton that was tracked last should always be greater than the previously tracked skeleton’s Id • We can use TrackingId to specify which skeletons are to be tracked or used by the application sensor.SkeletonStream.AppChoosesSkeletons = true; sensor.SkeletonStream.ChooseSkeletons(skeleton.TrackingId);

6. Monitoring Changes in the Skeleton • To track the development of a particular user, it is important to tell if the user has gone out of view and another user comes into view • The TrackingId can be used to monitor the changes in the skeleton being tracked (assuming only one user in view) • Use a local variable to store the TrackingId for the current skeleton • Update it when the skeleton ID is changed public int CurrentSkeletonID =0; if (skeleton != null && this.CurrentSkeletonID != skeleton.TrackingId) { this.CurrentSkeletonID = skeleton.TrackingId; }

7. Monitoring Changes in the Skeleton: A Use Case • Intrusion detector app: in skeleton frame event handler • Set TrackingId when a new skeleton is detected, and save an image, only track the skeleton • Continue tracking this skeleton without saving more images • When the current skeleton is no longer tracked, reset to the default tracking mode (SDK tracks all possible skeletons in view) public int CurrentSkeletonID =0; // member variable in MainApp class

8. Monitoring Changes in the Skeleton: A Use Case Skeleton skeleton; // local variable in event handler If(CurrentSkeletonID == 0) { skeleton = (from trackSkeleton in totalSkeleton where trackSkeleton.TrackingState == SkeletonTrackingState.Tracked select trackSkeleton).FirstOrDefault(); if (skeleton == null) { return; } CurrentSkeletonID = skeleton.TrackingId; this.sensor.SkeletonStream.AppChoosesSkeletons = true; this.sensor.SkeletonStream.ChooseSkeletons(CurrentSkeletonID); if (skeleton.Joints[JointType.Head].TrackingState == JointTrackingState.Tracked) { this.SaveImage(); } }

9. Monitoring Changes in the Skeleton: A Use Case if (CurrentSkeletonID != 0) // we are currently tracking a skeleton { skeleton = (from trackSkeleton in totalSkeleton where trackSkeleton.TrackingState == SkeletonTrackingState.Tracked && trackSkeleton.TrackingId == CurrentSkeletonID select trackSkeleton).FirstOrDefault(); if (skeleton == null) { // reset to default tracking mode CurrentSkeletonID = 0; this.sensor.SkeletonStream.AppChoosesSkeletons = false; } }

10. Joints and JointCollection • Each Skeleton object has a property named Joints, which is a type of JointCollection and contains all the traceable joints • JointCollecton contains a set of Joints and can be accessed by specifying the index value • When you pass JointType to get the Joint point, it will return the Joint object Skeleton skeleton = (from trackskeleton in totalSkeleton where trackskeleton.TrackingState == SkeletonTrackingState.Tracked select trackskeleton).FirstOrDefault(); Joint headJoint= skeleton.Joints[JointType.Head]);

11. The Joint structure has three properties JointType: 20 joints Positions: SkeletonPoint type (x,y,z) TrackingState: NotTracked, Tracked, Inferred Joint

12. Bones are the visual representation between joints The complete hierarchy of a skeleton is composed of a series of bones Each bone in a skeleton hierarchy has a parent joint and a child joint Every joint can be a parent and child joint unless it is a leaf joint Bones: Connecting Joints

13. Bone Sequence for a Default Skeleton • Root joint: Highest joint in hierarchy. Each skeleton has one root joint • For a default skeleton: Hip Center is the root joint

14. Bone Sequence for a Seated Skeleton • For seated skeleton: Shoulder Center is the root joint

15. Drawing bones between joints • Bones are the visual representation between joints and can be represented by a line or any other object void drawBone(Joint trackedJoint1, Joint trackedJoint2) { Line skeletonBone = new Line(); skeletonBone.Stroke = Brushes.Black; skeletonBone.StrokeThickness = 3; Point joint1 = this.ScalePosition(trackedJoint1.Position); skeletonBone.X1 = joint1.X; skeletonBone.Y1 = joint1.Y; Point joint2 = this.ScalePosition(trackedJoint2.Position); skeletonBone.X2 = joint2.X; skeletonBone.Y2 = joint2.Y; myCanvas.Children.Add(skeletonBone); }

16. ClippedEdges • ClippedEdges (of type FrameEdges) describes which parts of the skeleton are out of the Kinect's view: • None, Right, Left, Top, Bottom • Can be used to give user feedbacks on the position

17. Build TrackingSkeleton App • Create a new C# WPF project with name TrackingSkeleton • Add Microsoft.Kinect reference • Design GUI • Added WindowLoaded() method in xaml file • Adding code

18. GUI Design Image control Canvas TextBlock

19. Adding Code KinectSensor sensor; Skeleton[] totalSkeleton = new Skeleton[6]; WriteableBitmap colorBitmap; byte[] colorPixels; Skeleton skeleton; int currentSkeletonID = 0; • Add member variables: • WindowLoaded(): • Enable both ColorImageStream and SkeletonStream • Register event handler for both ColorFrameReady and SkeletonFrameReady events

20. Add Event Handler for skeleton frames void skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e) { canvas1.Children.Clear(); using (SkeletonFrame skeletonFrame = e.OpenSkeletonFrame()) { if (skeletonFrame == null) { return; } skeletonFrame.CopySkeletonDataTo(totalSkeleton); skeleton = (from trackskeleton in totalSkeleton where trackskeleton.TrackingState == SkeletonTrackingState.Tracked select trackskeleton).FirstOrDefault(); if (skeleton == null) return; if (skeleton != null && this.currentSkeletonID != skeleton.TrackingId) { this.currentSkeletonID = skeleton.TrackingId; int totalTrackedJoints = skeleton.Joints.Where(item => item.TrackingState == JointTrackingState.Tracked).Count(); string TrackedTime = DateTime.Now.ToString("hh:mm:ss"); string status = "Skeleton Id: " + this.currentSkeletonID + ", total tracked joints: " + totalTrackedJoints + ", TrackTime: " + TrackedTime+"\n"; this.textBlock1.Text += status; } DrawSkeleton(skeleton); } }}

21. Adding Code private void DrawSkeleton(Skeleton skeleton) { drawBone(skeleton.Joints[JointType.Head], skeleton.Joints[JointType.ShoulderCenter]); drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.Spine]); drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.ShoulderLeft]); drawBone(skeleton.Joints[JointType.ShoulderLeft], skeleton.Joints[JointType.ElbowLeft]); drawBone(skeleton.Joints[JointType.ElbowLeft], skeleton.Joints[JointType.WristLeft]); drawBone(skeleton.Joints[JointType.WristLeft], skeleton.Joints[JointType.HandLeft]); drawBone(skeleton.Joints[JointType.ShoulderCenter], skeleton.Joints[JointType.ShoulderRight]); drawBone(skeleton.Joints[JointType.ShoulderRight], skeleton.Joints[JointType.ElbowRight]); drawBone(skeleton.Joints[JointType.ElbowRight], skeleton.Joints[JointType.WristRight]); drawBone(skeleton.Joints[JointType.WristRight], skeleton.Joints[JointType.HandRight]); // more on next slide }

22. Adding Code private void DrawSkeleton(Skeleton skeleton) { // continue from previous slid drawBone(skeleton.Joints[JointType.Spine], skeleton.Joints[JointType.HipCenter]); drawBone(skeleton.Joints[JointType.HipCenter], skeleton.Joints[JointType.HipLeft]); drawBone(skeleton.Joints[JointType.HipLeft], skeleton.Joints[JointType.KneeLeft]); drawBone(skeleton.Joints[JointType.KneeLeft], skeleton.Joints[JointType.AnkleLeft]); drawBone(skeleton.Joints[JointType.AnkleLeft], skeleton.Joints[JointType.FootLeft]); drawBone(skeleton.Joints[JointType.HipCenter], skeleton.Joints[JointType.HipRight]); drawBone(skeleton.Joints[JointType.HipRight], skeleton.Joints[JointType.KneeRight]); drawBone(skeleton.Joints[JointType.KneeRight], skeleton.Joints[JointType.AnkleRight]); drawBone(skeleton.Joints[JointType.AnkleRight], skeleton.Joints[JointType.FootRight]); } private Point ScalePosition(SkeletonPoint skeletonPoint) { DepthImagePoint depthPoint = this.sensor.CoordinateMapper.MapSkeletonPointToDepthPoint(skeletonPoint, DepthImageFormat.Resolution320x240Fps30); return new Point(depthPoint.X, depthPoint.Y); }

23. Adding Code void drawBone(Joint trackedJoint1, Joint trackedJoint2) { Line bone = new Line(); bone.Stroke = Brushes.Red; bone.StrokeThickness = 3; Point joint1 = this.ScalePosition(trackedJoint1.Position); bone.X1 = joint1.X; bone.Y1 = joint1.Y; Point joint2 = this.ScalePosition(trackedJoint2.Position); bone.X2 = joint2.X; bone.Y2 = joint2.Y; canvas1.Children.Add(bone); }

24. Challenge Task • For advanced students, measure the angle formed between left/right arm and torso • Draw an arc between the arm and torso • Display the angle value with a TextBox on top of the arc EEC492/693/793 - iPhone Application Development

25. Helper Code for Advanced Task(https://dynamicdatadisplay.codeplex.com/discussions/76901/) private void AddCircularArcGraph(Point startPoint, Point endPoint, Size size) { PathFigure pf = new PathFigure(); pf.StartPoint = new Point(startPoint.X, startPoint.Y); ArcSegment arcSegment = new ArcSegment(); arcSegment.Point = new Point(endPoint.X, endPoint.Y); arcSegment.Size = size; arcSegment.SweepDirection = SweepDirection.Counterclockwise; PathSegmentCollection psc = new PathSegmentCollection(); psc.Add(arcSegment); pf.Segments = psc; PathFigureCollection pfc = new PathFigureCollection(); pfc.Add(pf); PathGeometry pg = new PathGeometry(); pg.Figures = pfc; var path = new System.Windows.Shapes.Path(); path.Stroke = Brushes.Black; path.StrokeThickness = 1; path.Data = pg; canvas1.Children.Add(path); }